[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:27 dist-test-slave-qgtk krb5kdc[21776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:27 dist-test-slave-qgtk krb5kdc[21776](info): set up 2 sockets
Jun 20 23:01:27 dist-test-slave-qgtk krb5kdc[21776](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 23:01:28.696382 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:28 dist-test-slave-qgtk krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:28.826407 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:33281
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:33281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:28.914647 21789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:28.914948 21789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:28.914978 21789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:28.916455 21789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:28.916524 21789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:28.916541 21789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:28.916558 21789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:28.916571 21789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:28.918478 21789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:33281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:28.918824 21789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:28.919119 21789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:28.925591 21797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:28.925635 21795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:28.925627 21789 server_base.cc:1034] running on GCE node
W20260620 23:01:28.925591 21794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:28.926263 21789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:28.926573 21789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:28.927757 21789 hybrid_clock.cc:648] HybridClock initialized: now 1781996488927747 us; error 35 us; skew 500 ppm
Jun 20 23:01:28 dist-test-slave-qgtk krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996488, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:28.930673 21789 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:28.931249 21789 webserver.cc:460] Webserver started at http://127.21.66.126:41211/ using document root <none> and password file <none>
I20260620 23:01:28.931557 21789 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:28.931607 21789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:28.931743 21789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:28.932992 21789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "34be9b7b2d4145d4bda431d1d4c188db"
format_stamp: "Formatted at 2026-06-20 23:01:28 on dist-test-slave-qgtk"
server_key: "2eb4833ed26b9404099d16892b00e2c4"
server_key_iv: "ca51ab767b9890326e0d70f53076b9c9"
server_key_version: "encryptionkey@0"
I20260620 23:01:28.933533 21789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "34be9b7b2d4145d4bda431d1d4c188db"
format_stamp: "Formatted at 2026-06-20 23:01:28 on dist-test-slave-qgtk"
server_key: "2eb4833ed26b9404099d16892b00e2c4"
server_key_iv: "ca51ab767b9890326e0d70f53076b9c9"
server_key_version: "encryptionkey@0"
I20260620 23:01:28.935184 21789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 23:01:28.936282 21804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:28.936702 21789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:28.936846 21789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "34be9b7b2d4145d4bda431d1d4c188db"
format_stamp: "Formatted at 2026-06-20 23:01:28 on dist-test-slave-qgtk"
server_key: "2eb4833ed26b9404099d16892b00e2c4"
server_key_iv: "ca51ab767b9890326e0d70f53076b9c9"
server_key_version: "encryptionkey@0"
I20260620 23:01:28.936945 21789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:28.951371 21789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:28.954267 21789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:28.954566 21789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:28.959617 21789 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:33281
I20260620 23:01:28.959656 21856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:33281 every 8 connection(s)
I20260620 23:01:28.960049 21789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:28.961082 21857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:28.962417 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 21789
I20260620 23:01:28.962584 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:28.963449 21769 external_mini_cluster.cc:1428] Setting key 049ea914f841be2e23b73ca3012ac8ee
I20260620 23:01:28.964290 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db: Bootstrap starting.
I20260620 23:01:28.965247 21857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:28.965595 21857 log.cc:824] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:28.966451 21857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db: No bootstrap required, opened a new log
I20260620 23:01:28.968403 21857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34be9b7b2d4145d4bda431d1d4c188db" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33281 } }
I20260620 23:01:28.968603 21857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:28.968655 21857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34be9b7b2d4145d4bda431d1d4c188db, State: Initialized, Role: FOLLOWER
I20260620 23:01:28.968825 21857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34be9b7b2d4145d4bda431d1d4c188db" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33281 } }
I20260620 23:01:28.968924 21857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:28.968976 21857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:28.969055 21857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:28.970033 21857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34be9b7b2d4145d4bda431d1d4c188db" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33281 } }
I20260620 23:01:28.970191 21857 leader_election.cc:302] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34be9b7b2d4145d4bda431d1d4c188db; no voters:
I20260620 23:01:28.970405 21857 leader_election.cc:288] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:28.970465 21862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:28.970642 21862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [term 1 LEADER]: Becoming Leader. State: Replica: 34be9b7b2d4145d4bda431d1d4c188db, State: Running, Role: LEADER
I20260620 23:01:28.970719 21857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:28.970724 21862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34be9b7b2d4145d4bda431d1d4c188db" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33281 } }
I20260620 23:01:28.971136 21863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34be9b7b2d4145d4bda431d1d4c188db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34be9b7b2d4145d4bda431d1d4c188db" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33281 } } }
I20260620 23:01:28.971235 21863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:28.971402 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34be9b7b2d4145d4bda431d1d4c188db. Latest consensus state: current_term: 1 leader_uuid: "34be9b7b2d4145d4bda431d1d4c188db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34be9b7b2d4145d4bda431d1d4c188db" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33281 } } }
I20260620 23:01:28.971483 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:28.972577 21867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:28.973062 21867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:28.974843 21867 catalog_manager.cc:1347] Generated new cluster ID: 32c5a0754ce34067bdebed7733651362
I20260620 23:01:28.974906 21867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:28 dist-test-slave-qgtk krb5kdc[21776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:28.984275 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:28.964602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.21.66.126:33281)
0620 23:01:28.964835 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:28.964846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:28.965074 (+ 228us) server_negotiation.cc:407] Connection header received
0620 23:01:28.968374 (+ 3300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:28.968382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:28.968473 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:28.968698 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:28.970588 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:28.972225 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:28.973860 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:28.974539 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:28.974672 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:28.979585 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:28.979591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:28.979591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:28.979606 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:28.981824 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:28.982473 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:28.982476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:28.982477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:28.982622 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:28.982962 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:28.982963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:28.982963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:28.983303 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:28.983484 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:28.983805 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:01:28.983967 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260620 23:01:28.985591 21867 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:28.986348 21867 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:28.996630 21867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34be9b7b2d4145d4bda431d1d4c188db: Generated new TSK 0
I20260620 23:01:28.996881 21867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:29.030309 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:33281
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:29.114934 21883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:29.115157 21883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:29.115183 21883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:29.116341 21883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:29.116385 21883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:29.116415 21883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:29.117879 21883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:33281
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:29.118099 21883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:29.118392 21883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:29.123884 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:29.123925 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:29.123890 21891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:29.124146 21883 server_base.cc:1034] running on GCE node
I20260620 23:01:29.124433 21883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:29.124747 21883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:29.125934 21883 hybrid_clock.cc:648] HybridClock initialized: now 1781996489125894 us; error 56 us; skew 500 ppm
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:29.128917 21883 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:29.129487 21883 webserver.cc:460] Webserver started at http://127.21.66.65:40191/ using document root <none> and password file <none>
I20260620 23:01:29.129733 21883 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:29.129803 21883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:29.129959 21883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:29.131213 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "e8c7ea44e8ad47849528e57b17fe9d89"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "30ea0d593147ed562d6497f7e0d90ecf"
server_key_iv: "0844c0667947752eb7c760f21ee86387"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.131618 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "e8c7ea44e8ad47849528e57b17fe9d89"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "30ea0d593147ed562d6497f7e0d90ecf"
server_key_iv: "0844c0667947752eb7c760f21ee86387"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.133030 21883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:01:29.133935 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.134235 21883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:29.134346 21883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "e8c7ea44e8ad47849528e57b17fe9d89"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "30ea0d593147ed562d6497f7e0d90ecf"
server_key_iv: "0844c0667947752eb7c760f21ee86387"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.134443 21883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:29.152143 21883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:29.155102 21883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:29.155325 21883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:29.155624 21883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:29.155951 21883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:29.156008 21883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.156054 21883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:29.156085 21883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.162544 21883 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:41539
I20260620 23:01:29.162603 22011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:41539 every 8 connection(s)
I20260620 23:01:29.162997 21883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:29.165577 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 21883
I20260620 23:01:29.165701 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:29.165855 21769 external_mini_cluster.cc:1428] Setting key 1ac027731b6dc77c074ebdddcaf324e5
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:01:29.175954 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.163981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:49715 (local address 127.21.66.126:33281)
0620 23:01:29.164042 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:29.164045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:29.164202 (+ 157us) server_negotiation.cc:407] Connection header received
0620 23:01:29.164794 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:29.164796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:29.164813 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:29.164892 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:29.166072 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.166734 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.168334 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.168696 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.168795 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.171785 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:29.171787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:29.171788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:29.171800 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:29.174172 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.174682 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.174684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.174684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.174766 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.175110 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.175111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.175111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.175305 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:29.175391 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:29.175691 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 23:01:29.175821 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:01:29.176105 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.163806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:33281 (local address 127.21.66.65:49715)
0620 23:01:29.164061 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:29.164104 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:29.164663 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:29.164991 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:29.164995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:29.165619 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:29.165906 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:29.165912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.166831 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.166832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:29.168208 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:29.168212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.168817 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.168818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:29.168935 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.169586 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:29.169598 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:29.171660 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:29.174286 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:29.174297 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:29.174300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:29.174581 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:29.174881 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:29.174882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:29.174883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:29.175024 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:29.175387 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:29.175392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:29.175541 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 23:01:29.175635 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:01:29.175854 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260620 23:01:29.176867 22012 heartbeater.cc:347] Connected to a master server at 127.21.66.126:33281
I20260620 23:01:29.177052 22012 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:29.177263 22012 heartbeater.cc:511] Master 127.21.66.126:33281 requested a full tablet report, sending...
I20260620 23:01:29.177973 21821 ts_manager.cc:194] Registered new tserver with Master: e8c7ea44e8ad47849528e57b17fe9d89 (127.21.66.65:41539)
I20260620 23:01:29.178761 21821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:49715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:29.197989 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:33281
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:29.290920 22017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:29.291165 22017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:29.291193 22017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:29.292528 22017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:29.292589 22017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:29.292620 22017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:29.294278 22017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:33281
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:29.294567 22017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:29.294853 22017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:29.300510 22023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:29.300510 22025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:29.300527 22022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:29.300529 22017 server_base.cc:1034] running on GCE node
I20260620 23:01:29.300966 22017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:29.301321 22017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:29.302498 22017 hybrid_clock.cc:648] HybridClock initialized: now 1781996489302492 us; error 39 us; skew 500 ppm
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:29.305454 22017 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:29.305975 22017 webserver.cc:460] Webserver started at http://127.21.66.66:36001/ using document root <none> and password file <none>
I20260620 23:01:29.306290 22017 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:29.306363 22017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:29.306543 22017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:29.307691 22017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "172a3348708c483e9350cfea6b3f514f"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "3b8e15f444243f7a1c4978d0727c3da6"
server_key_iv: "6f2acfdb88098aee86498ecceb41ce42"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.308081 22017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "172a3348708c483e9350cfea6b3f514f"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "3b8e15f444243f7a1c4978d0727c3da6"
server_key_iv: "6f2acfdb88098aee86498ecceb41ce42"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.309589 22017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:29.310549 22032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.310894 22017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:29.311092 22017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "172a3348708c483e9350cfea6b3f514f"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "3b8e15f444243f7a1c4978d0727c3da6"
server_key_iv: "6f2acfdb88098aee86498ecceb41ce42"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.311195 22017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:29.323118 22017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:29.325788 22017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:29.325989 22017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:29.326314 22017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:29.326683 22017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:29.326741 22017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.326776 22017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:29.326799 22017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.333101 22017 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:44011
I20260620 23:01:29.333130 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:44011 every 8 connection(s)
I20260620 23:01:29.333515 22017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:01:29.333683 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22017
I20260620 23:01:29.333779 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:29.333992 21769 external_mini_cluster.cc:1428] Setting key 11a43fde6e0e1550366352fa5856178c
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
I20260620 23:01:29.345903 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.334477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:33281 (local address 127.21.66.66:54905)
0620 23:01:29.334651 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:29.334697 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:29.335444 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:29.335848 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:29.335852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:29.336701 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:29.336922 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:29.336927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.337768 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.337769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:29.339229 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:29.339233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.339656 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.339658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:29.339821 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.340394 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:29.340406 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:29.342336 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:29.344156 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:29.344166 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:29.344169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:29.344442 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:29.344748 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:29.344749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:29.344750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:29.344925 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:29.345233 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:29.345239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:29.345396 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 23:01:29.345495 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:01:29.345714 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260620 23:01:29.345896 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.334691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:54905 (local address 127.21.66.126:33281)
0620 23:01:29.334736 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:29.334739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:29.334773 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:01:29.335665 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:29.335667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:29.335687 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:29.335743 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:29.337024 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.337653 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.339344 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.339556 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.339636 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.342446 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:29.342448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:29.342448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:29.342460 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:29.344015 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.344581 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.344582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.344583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.344645 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.345005 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.345006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.345006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.345152 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:29.345210 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:29.345580 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 23:01:29.345720 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
I20260620 23:01:29.346664 22146 heartbeater.cc:347] Connected to a master server at 127.21.66.126:33281
I20260620 23:01:29.346880 22146 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:29.347105 22146 heartbeater.cc:511] Master 127.21.66.126:33281 requested a full tablet report, sending...
I20260620 23:01:29.347627 21821 ts_manager.cc:194] Registered new tserver with Master: 172a3348708c483e9350cfea6b3f514f (127.21.66.66:44011)
I20260620 23:01:29.348237 21821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:54905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:29.366873 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:33281
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:29.458534 22151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:29.458819 22151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:29.458868 22151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:29.460264 22151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:29.460345 22151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:29.460397 22151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:29.462248 22151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:33281
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:29.462558 22151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:29.462828 22151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:29.469134 22156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:29.469134 22157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:29.469377 22151 server_base.cc:1034] running on GCE node
W20260620 23:01:29.469482 22159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:29.469692 22151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:29.470031 22151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:29.471237 22151 hybrid_clock.cc:648] HybridClock initialized: now 1781996489471205 us; error 46 us; skew 500 ppm
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:29.474428 22151 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:29.474920 22151 webserver.cc:460] Webserver started at http://127.21.66.67:34459/ using document root <none> and password file <none>
I20260620 23:01:29.475181 22151 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:29.475234 22151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:29.475389 22151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:29.476532 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "00b567588f304bf486dd5ed352730b78"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "825c78d4d702c9c74739adcd814fd8d9"
server_key_iv: "9da73811cc55ea07ac4b4ea59d1b0107"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.476928 22151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "00b567588f304bf486dd5ed352730b78"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "825c78d4d702c9c74739adcd814fd8d9"
server_key_iv: "9da73811cc55ea07ac4b4ea59d1b0107"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.478539 22151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:29.479489 22166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.479722 22151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:29.479822 22151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "00b567588f304bf486dd5ed352730b78"
format_stamp: "Formatted at 2026-06-20 23:01:29 on dist-test-slave-qgtk"
server_key: "825c78d4d702c9c74739adcd814fd8d9"
server_key_iv: "9da73811cc55ea07ac4b4ea59d1b0107"
server_key_version: "encryptionkey@0"
I20260620 23:01:29.479915 22151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:29.503852 22151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:29.506273 22151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:29.506400 22151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:29.506626 22151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:29.506973 22151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:29.507011 22151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.507066 22151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:29.507102 22151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:29.513384 22151 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:37935
I20260620 23:01:29.513468 22279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:37935 every 8 connection(s)
I20260620 23:01:29.513821 22151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:29.523146 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22151
I20260620 23:01:29.523234 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:29.523423 21769 external_mini_cluster.cc:1428] Setting key a87652fefd28e3ed6d1387e7ab65f2f3
I20260620 23:01:29.525890 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.514918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:55817 (local address 127.21.66.126:33281)
0620 23:01:29.514998 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:29.515001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:29.515225 (+ 224us) server_negotiation.cc:407] Connection header received
0620 23:01:29.515821 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:29.515823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:29.515835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:29.515914 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:29.516950 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.517463 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.518832 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.519202 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.519329 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.522436 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:29.522438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:29.522439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:29.522451 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:29.524334 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.525016 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.525017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.525017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.525087 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.525296 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.525296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.525297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.525408 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:29.525457 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:29.525687 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 23:01:29.525821 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:01:29.525998 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.514923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:33281 (local address 127.21.66.67:55817)
0620 23:01:29.515091 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:29.515132 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:29.515654 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:29.516011 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:29.516018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:29.516627 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:29.516839 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:29.516845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.517564 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.517565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:29.518686 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:29.518690 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.519418 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.519420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:29.519568 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.520198 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:29.520207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:29.521985 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:29.524588 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:29.524598 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:29.524601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:29.524931 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:29.525139 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:29.525140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:29.525140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:29.525254 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:29.525448 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:29.525450 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:29.525574 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 23:01:29.525661 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:01:29.525792 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260620 23:01:29.526561 22280 heartbeater.cc:347] Connected to a master server at 127.21.66.126:33281
I20260620 23:01:29.526666 22280 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:29.526827 22280 heartbeater.cc:511] Master 127.21.66.126:33281 requested a full tablet report, sending...
I20260620 23:01:29.527217 21821 ts_manager.cc:194] Registered new tserver with Master: 00b567588f304bf486dd5ed352730b78 (127.21.66.67:37935)
I20260620 23:01:29.527560 21821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:55817
I20260620 23:01:29.536339 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:29.561637 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.552896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.21.66.65:41539)
0620 23:01:29.553116 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:29.553122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:29.553135 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:01:29.553163 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:29.553166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:29.553180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:29.553251 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:29.553930 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.554785 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.555565 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.555976 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.556038 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.558658 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:29.558677 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:29.558677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:29.558687 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:29.560445 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.560863 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.560866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.560866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.560937 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.561117 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.561118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.561118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.561271 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:29.561384 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:29.561479 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 23:01:29.561552 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":124,"threads_started":1}
W20260620 23:01:29.562206 21926 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36688
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[21776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:29.587175 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:29.579217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.21.66.65:41539)
0620 23:01:29.579317 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:29.579324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:29.579390 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:01:29.579445 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:29.579446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:29.579457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:29.579510 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:29.580116 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.580816 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.581603 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:29.581889 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:29.581952 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:29.584524 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:29.584526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:29.584526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:29.584533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:29.585861 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.586286 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.586288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.586288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.586351 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:29.586564 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:29.586564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:29.586565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:29.586663 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:29.586841 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:29.587025 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 23:01:29.587103 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:01:29.587932 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 21883
I20260620 23:01:29.593098 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22017
I20260620 23:01:29.598754 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22151
I20260620 23:01:29.603775 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 21789
2026-06-20T23:01:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2010 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[22294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[22294](info): set up 2 sockets
Jun 20 23:01:29 dist-test-slave-qgtk krb5kdc[22294](info): commencing operation
krb5kdc: starting...
W20260620 23:01:30.690073 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:30 dist-test-slave-qgtk krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:30.827158 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34633
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:30.917083 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:30.917353 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:30.917382 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:30.918900 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:30.918962 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:30.918978 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:30.918993 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:30.919008 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:30.920940 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:30.921272 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:30.921553 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:01:30.928047 22307 server_base.cc:1034] running on GCE node
W20260620 23:01:30.928171 22313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:30.927990 22312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:30.927959 22315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:30.928566 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:30.928917 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:30.930090 22307 hybrid_clock.cc:648] HybridClock initialized: now 1781996490930040 us; error 67 us; skew 500 ppm
Jun 20 23:01:30 dist-test-slave-qgtk krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996490, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:30.933146 22307 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:30.933656 22307 webserver.cc:460] Webserver started at http://127.21.66.126:37605/ using document root <none> and password file <none>
I20260620 23:01:30.933907 22307 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:30.933957 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:30.934098 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:30.935293 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "75be126092e7489a8e92e0536a666ff0"
format_stamp: "Formatted at 2026-06-20 23:01:30 on dist-test-slave-qgtk"
server_key: "59b907261ec93367b8fed6fa145341c2"
server_key_iv: "536b513d7fc31e15fcf51079622268fe"
server_key_version: "encryptionkey@0"
I20260620 23:01:30.935683 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "75be126092e7489a8e92e0536a666ff0"
format_stamp: "Formatted at 2026-06-20 23:01:30 on dist-test-slave-qgtk"
server_key: "59b907261ec93367b8fed6fa145341c2"
server_key_iv: "536b513d7fc31e15fcf51079622268fe"
server_key_version: "encryptionkey@0"
I20260620 23:01:30.937196 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:30.938284 22322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:30.938472 22307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:30.938628 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "75be126092e7489a8e92e0536a666ff0"
format_stamp: "Formatted at 2026-06-20 23:01:30 on dist-test-slave-qgtk"
server_key: "59b907261ec93367b8fed6fa145341c2"
server_key_iv: "536b513d7fc31e15fcf51079622268fe"
server_key_version: "encryptionkey@0"
I20260620 23:01:30.938709 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:30.954638 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:30.957113 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:30.957235 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:30.961728 22307 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:34633
I20260620 23:01:30.961800 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:34633 every 8 connection(s)
I20260620 23:01:30.962193 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:30.963203 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:30.963284 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22307
I20260620 23:01:30.963487 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:30.963706 21769 external_mini_cluster.cc:1428] Setting key 73932d0c34e3194d92d4fcd03e796be8
I20260620 23:01:30.967005 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0: Bootstrap starting.
I20260620 23:01:30.967746 22375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:30.968057 22375 log.cc:824] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:30.968990 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0: No bootstrap required, opened a new log
Jun 20 23:01:30 dist-test-slave-qgtk krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:30.971273 22375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75be126092e7489a8e92e0536a666ff0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34633 } }
I20260620 23:01:30.971498 22375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:30.971565 22375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75be126092e7489a8e92e0536a666ff0, State: Initialized, Role: FOLLOWER
I20260620 23:01:30.971704 22375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75be126092e7489a8e92e0536a666ff0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34633 } }
I20260620 23:01:30.971776 22375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:30.971799 22375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:30.971869 22375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:30.972615 22375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75be126092e7489a8e92e0536a666ff0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34633 } }
I20260620 23:01:30.972708 22375 leader_election.cc:302] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75be126092e7489a8e92e0536a666ff0; no voters:
I20260620 23:01:30.972931 22375 leader_election.cc:288] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:30.973050 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:30.973222 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [term 1 LEADER]: Becoming Leader. State: Replica: 75be126092e7489a8e92e0536a666ff0, State: Running, Role: LEADER
I20260620 23:01:30.973244 22375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:30.973335 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75be126092e7489a8e92e0536a666ff0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34633 } }
I20260620 23:01:30.973738 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75be126092e7489a8e92e0536a666ff0. Latest consensus state: current_term: 1 leader_uuid: "75be126092e7489a8e92e0536a666ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75be126092e7489a8e92e0536a666ff0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34633 } } }
I20260620 23:01:30.973742 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75be126092e7489a8e92e0536a666ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75be126092e7489a8e92e0536a666ff0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34633 } } }
I20260620 23:01:30.973819 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:30.973826 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0 [sys.catalog]: This master's current role is: LEADER
W20260620 23:01:30.975116 22396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:30.975165 22396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:30.975240 22389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:30.975389 22389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:30.976473 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:30.964331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.21.66.126:34633)
0620 23:01:30.964527 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:30.964538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:30.964573 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:30.965102 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:30.965112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:30.965183 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:30.965343 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:30.965952 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:30.967396 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:30.968270 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:30.968883 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:30.969001 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:30.972001 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:30.972005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:30.972006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:30.972017 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:30.974548 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:30.975049 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:30.975053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:30.975053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:30.975164 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:30.975513 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:30.975514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:30.975514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:30.975835 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:30.976012 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:30.976131 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:01:30.976276 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260620 23:01:30.977046 22389 catalog_manager.cc:1347] Generated new cluster ID: 845a40a2779045449df01db23eb4cd5e
I20260620 23:01:30.977098 22389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:30.996582 22389 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:30.997390 22389 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:31.002420 22389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75be126092e7489a8e92e0536a666ff0: Generated new TSK 0
I20260620 23:01:31.002664 22389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:31.041122 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34633
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:31.127460 22401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:31.127705 22401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:31.127735 22401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:31.129051 22401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:31.129108 22401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:31.129139 22401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:31.130995 22401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:31.131292 22401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:31.131564 22401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:31.137040 22407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:31.137027 22406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:31.137264 22409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:31.137610 22401 server_base.cc:1034] running on GCE node
I20260620 23:01:31.137883 22401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:31.138104 22401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:31.139330 22401 hybrid_clock.cc:648] HybridClock initialized: now 1781996491139312 us; error 33 us; skew 500 ppm
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:31.142421 22401 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:31.143093 22401 webserver.cc:460] Webserver started at http://127.21.66.65:37313/ using document root <none> and password file <none>
I20260620 23:01:31.143350 22401 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:31.143424 22401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:31.143587 22401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:31.144724 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "ee3da2defc2d466b8600c9f7050b2d1c"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "830aedc2697ef4314ec458c96781eeea"
server_key_iv: "c96d5ae2f90192b63dd7f0b812996165"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.145115 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "ee3da2defc2d466b8600c9f7050b2d1c"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "830aedc2697ef4314ec458c96781eeea"
server_key_iv: "c96d5ae2f90192b63dd7f0b812996165"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.146714 22401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:31.147737 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.147987 22401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:31.148119 22401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "ee3da2defc2d466b8600c9f7050b2d1c"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "830aedc2697ef4314ec458c96781eeea"
server_key_iv: "c96d5ae2f90192b63dd7f0b812996165"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.148211 22401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:31.168546 22401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:31.171519 22401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:31.171717 22401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:31.172000 22401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:31.172354 22401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:31.172426 22401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.172489 22401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:31.172523 22401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.179454 22401 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:42879
I20260620 23:01:31.179492 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:42879 every 8 connection(s)
I20260620 23:01:31.179885 22401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:31.186956 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22401
I20260620 23:01:31.187077 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:31.187259 21769 external_mini_cluster.cc:1428] Setting key a920c7e84354de1b64ee72e34dabc4c0
I20260620 23:01:31.191134 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.180754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:53573 (local address 127.21.66.126:34633)
0620 23:01:31.180843 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.180846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.181010 (+ 164us) server_negotiation.cc:407] Connection header received
0620 23:01:31.182094 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.182096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.182107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.182244 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:31.182952 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.183467 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.184618 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.184864 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.184940 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.187713 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:31.187714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:31.187714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:31.187723 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:31.189439 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.189944 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.189946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.189947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.190013 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.190328 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.190329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.190330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.190464 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:31.190591 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.190915 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.191017 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:01:31.191380 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.180699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34633 (local address 127.21.66.65:53573)
0620 23:01:31.180868 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.180908 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.181961 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.182312 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.182316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.182612 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:31.182823 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.182827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.183599 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.183600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.184512 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.184515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.185055 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.185057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.185187 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.185769 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:31.185778 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:31.187613 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:31.189575 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:31.189585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:31.189588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:31.189858 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:31.190085 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:31.190086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:31.190086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:31.190257 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:31.190603 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:31.190611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:31.190772 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.190894 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.191107 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260620 23:01:31.191872 22530 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34633
I20260620 23:01:31.191982 22530 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:31.192202 22530 heartbeater.cc:511] Master 127.21.66.126:34633 requested a full tablet report, sending...
I20260620 23:01:31.192878 22339 ts_manager.cc:194] Registered new tserver with Master: ee3da2defc2d466b8600c9f7050b2d1c (127.21.66.65:42879)
I20260620 23:01:31.193921 22339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:53573
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:31.218098 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34633
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:31.307214 22535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:31.307510 22535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:31.307559 22535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:31.308807 22535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:31.308880 22535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:31.308929 22535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:31.310539 22535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:31.310835 22535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:31.311082 22535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:31.316404 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:31.316404 22543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:31.316407 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:31.316700 22535 server_base.cc:1034] running on GCE node
I20260620 23:01:31.317016 22535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:31.317299 22535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:31.318462 22535 hybrid_clock.cc:648] HybridClock initialized: now 1781996491318444 us; error 41 us; skew 500 ppm
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:31.321409 22535 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:31.321883 22535 webserver.cc:460] Webserver started at http://127.21.66.66:36023/ using document root <none> and password file <none>
I20260620 23:01:31.322099 22535 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:31.322218 22535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:31.322355 22535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:31.323534 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "cfb0967483304340be28c3c654342eef"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "b2a07475127bb4373b7ad7985b6f6491"
server_key_iv: "0a4e46b72ad122ce482219c83d1f186a"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.323873 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "cfb0967483304340be28c3c654342eef"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "b2a07475127bb4373b7ad7985b6f6491"
server_key_iv: "0a4e46b72ad122ce482219c83d1f186a"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.325299 22535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:31.326267 22550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.326435 22535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:31.326525 22535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "cfb0967483304340be28c3c654342eef"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "b2a07475127bb4373b7ad7985b6f6491"
server_key_iv: "0a4e46b72ad122ce482219c83d1f186a"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.326633 22535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:31.334419 22535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:31.336673 22535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:31.336800 22535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:31.336992 22535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:31.337337 22535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:31.337379 22535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.337404 22535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:31.337416 22535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.343487 22535 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:46089
I20260620 23:01:31.343551 22663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:46089 every 8 connection(s)
I20260620 23:01:31.343963 22535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:31.353819 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22535
I20260620 23:01:31.353907 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:31.354038 21769 external_mini_cluster.cc:1428] Setting key 988a5e5f38519e1d1150fdb271454ebb
I20260620 23:01:31.358662 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.344923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:37819 (local address 127.21.66.126:34633)
0620 23:01:31.345010 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.345012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.345273 (+ 261us) server_negotiation.cc:407] Connection header received
0620 23:01:31.346445 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.346446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.346458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.346526 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:31.347306 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.347852 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.349046 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.349283 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.349345 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.352093 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:31.352095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:31.352095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:31.352103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:31.356881 (+ 4778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.357429 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.357431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.357431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.357511 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.357889 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.357890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.357891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.358036 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:31.358116 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.358458 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.358555 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:01:31.358892 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.344877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34633 (local address 127.21.66.66:37819)
0620 23:01:31.345090 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.345137 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.346337 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.346608 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.346612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.346986 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:31.347206 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.347211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.347948 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.347949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.348951 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.348953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.349372 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.349374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.349497 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.350042 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:31.350056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:31.351979 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:31.356993 (+ 5014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:31.357004 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:31.357007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:31.357339 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:31.357610 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:31.357611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:31.357611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:31.357786 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:31.358143 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:31.358152 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:31.358307 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.358423 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.358618 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260620 23:01:31.359496 22664 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34633
I20260620 23:01:31.359627 22664 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:31.359866 22664 heartbeater.cc:511] Master 127.21.66.126:34633 requested a full tablet report, sending...
I20260620 23:01:31.360345 22339 ts_manager.cc:194] Registered new tserver with Master: cfb0967483304340be28c3c654342eef (127.21.66.66:46089)
I20260620 23:01:31.360808 22339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:37819
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:31.387429 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34633
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:37959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:31.477602 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:31.477840 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:31.477869 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:31.479157 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:31.479209 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:31.479239 22669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:31.480837 22669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:37959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:31.481053 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:31.481251 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:31.486712 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:31.486712 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:31.486815 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:31.486970 22669 server_base.cc:1034] running on GCE node
I20260620 23:01:31.487185 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:31.487404 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:31.488550 22669 hybrid_clock.cc:648] HybridClock initialized: now 1781996491488535 us; error 29 us; skew 500 ppm
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:31.491607 22669 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:31.492129 22669 webserver.cc:460] Webserver started at http://127.21.66.67:34389/ using document root <none> and password file <none>
I20260620 23:01:31.492354 22669 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:31.492398 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:31.492501 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:31.493770 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "f79979e523444e29a73446edf300238e"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "7c95bcca02fc057a62a82438fb9ad110"
server_key_iv: "16b4fe122ef985f59fec4fcddcc67a8c"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.494263 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "f79979e523444e29a73446edf300238e"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "7c95bcca02fc057a62a82438fb9ad110"
server_key_iv: "16b4fe122ef985f59fec4fcddcc67a8c"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.496208 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:31.497149 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.497416 22669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:31.497499 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "f79979e523444e29a73446edf300238e"
format_stamp: "Formatted at 2026-06-20 23:01:31 on dist-test-slave-qgtk"
server_key: "7c95bcca02fc057a62a82438fb9ad110"
server_key_iv: "16b4fe122ef985f59fec4fcddcc67a8c"
server_key_version: "encryptionkey@0"
I20260620 23:01:31.497586 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:31.519970 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:31.522286 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:31.522455 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:31.522707 22669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:31.523077 22669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:31.523132 22669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.523173 22669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:31.523201 22669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:31.529246 22669 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:35041
I20260620 23:01:31.529289 22797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:35041 every 8 connection(s)
I20260620 23:01:31.529698 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:01:31.533826 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22669
I20260620 23:01:31.533954 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:31.534169 21769 external_mini_cluster.cc:1428] Setting key 56bf96e028d62f5048820e12d1b0fb3a
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:31.540798 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.530547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:44361 (local address 127.21.66.126:34633)
0620 23:01:31.530603 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.530605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.530861 (+ 256us) server_negotiation.cc:407] Connection header received
0620 23:01:31.531805 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.531806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.531819 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.531881 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:31.532520 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.533291 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.534712 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.534978 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.535067 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.537449 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:31.537451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:31.537451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:31.537461 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:31.539139 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.539667 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.539669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.539669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.539738 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.540025 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.540026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.540026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.540174 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:31.540287 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.540621 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.540695 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:01:31.541118 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.530490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34633 (local address 127.21.66.67:44361)
0620 23:01:31.530652 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.530690 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.531678 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.531941 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.531945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.532216 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:31.532421 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.532426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.533426 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.533428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.534619 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.534622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.535083 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.535084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.535230 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.535741 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:31.535749 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:31.537325 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:31.539280 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:31.539288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:31.539290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:31.539590 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:31.539808 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:31.539808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:31.539809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:31.539918 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:31.540296 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:31.540310 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:31.540456 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.540565 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.540799 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 23:01:31.541863 22798 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34633
I20260620 23:01:31.541973 22798 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:31.542202 22798 heartbeater.cc:511] Master 127.21.66.126:34633 requested a full tablet report, sending...
I20260620 23:01:31.542655 22339 ts_manager.cc:194] Registered new tserver with Master: f79979e523444e29a73446edf300238e (127.21.66.67:35041)
I20260620 23:01:31.543046 22339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:44361
I20260620 23:01:31.546278 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:01:31.554395 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.547882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.21.66.126:34633)
0620 23:01:31.547961 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.547963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.547998 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:31.548054 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.548055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.548065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.548124 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:31.548701 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.549287 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.550073 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.550369 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.550431 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.551841 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:31.551842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:31.551842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:31.551849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:31.553096 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.553481 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.553482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.553482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.553549 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.553722 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.553723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.553723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.553893 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:31.553975 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.554193 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.554274 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:01:31.557017 22339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:31.558871 22339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:31.565797 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.561641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.21.66.67:35041)
0620 23:01:31.562155 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.562160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.562170 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:31.562210 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.562212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.562237 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.562320 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:31.562527 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.563320 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.564571 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.565343 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.565425 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.565489 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.565659 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.565700 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":82,"threads_started":1}
I20260620 23:01:31.565805 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.561525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:35041 (local address 127.0.0.1:58894)
0620 23:01:31.561683 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.561726 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.561791 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.562337 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.562341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.562344 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:31.562433 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.562436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.563439 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.563440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.564473 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.564475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.565444 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.565445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.565537 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.565552 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.565636 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.565692 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260620 23:01:31.566401 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.561802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42879 (local address 127.0.0.1:35538)
0620 23:01:31.562036 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.562046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.562094 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.562638 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.562639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.562641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:31.562678 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.562679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.564056 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.564057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.565161 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.565163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.566150 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.566151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.566220 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.566224 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.566284 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.566329 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":72,"threads_started":1}
I20260620 23:01:31.566416 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.562033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.21.66.65:42879)
0620 23:01:31.562259 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.562263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.562271 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:31.562452 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.562455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.562475 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.562548 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:31.562772 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.563953 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.565262 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.566036 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.566165 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.566225 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.566298 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.566340 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":105,"threads_started":1}
I20260620 23:01:31.566547 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.561525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:46089 (local address 127.0.0.1:46026)
0620 23:01:31.561916 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.561927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.561993 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.562341 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.562342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.562345 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:31.562440 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.562443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.563942 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.563943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.565099 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.565101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.566270 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.566271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.566362 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.566366 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.566423 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.566472 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":61,"thread_start_us":91,"threads_started":1}
I20260620 23:01:31.567047 22732 tablet_service.cc:1467] Processing CreateTablet for tablet 02bbac8b2eeb4bffa0257e5ac0b720dc (DEFAULT_TABLE table=test-table [id=785489bedd8b4c8fb17ffe14eaa3d83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:31.567047 22464 tablet_service.cc:1467] Processing CreateTablet for tablet 02bbac8b2eeb4bffa0257e5ac0b720dc (DEFAULT_TABLE table=test-table [id=785489bedd8b4c8fb17ffe14eaa3d83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:31.567188 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.561727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.21.66.66:46089)
0620 23:01:31.561912 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.561918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.561978 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:01:31.562068 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.562070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.562089 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.562203 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:31.562539 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.563810 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.565208 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.566065 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.566168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.566234 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.567073 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.567125 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260620 23:01:31.567412 22732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bbac8b2eeb4bffa0257e5ac0b720dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:31.567412 22464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bbac8b2eeb4bffa0257e5ac0b720dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:31.567860 22598 tablet_service.cc:1467] Processing CreateTablet for tablet 02bbac8b2eeb4bffa0257e5ac0b720dc (DEFAULT_TABLE table=test-table [id=785489bedd8b4c8fb17ffe14eaa3d83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:31.568183 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02bbac8b2eeb4bffa0257e5ac0b720dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:31.570889 22815 tablet_bootstrap.cc:492] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c: Bootstrap starting.
I20260620 23:01:31.571236 22817 tablet_bootstrap.cc:492] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef: Bootstrap starting.
I20260620 23:01:31.571732 22815 tablet_bootstrap.cc:654] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:31.571823 22817 tablet_bootstrap.cc:654] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:31.572278 22817 log.cc:824] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:31.572286 22815 log.cc:824] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:31.572697 22816 tablet_bootstrap.cc:492] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e: Bootstrap starting.
I20260620 23:01:31.573143 22817 tablet_bootstrap.cc:492] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef: No bootstrap required, opened a new log
I20260620 23:01:31.573169 22815 tablet_bootstrap.cc:492] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c: No bootstrap required, opened a new log
I20260620 23:01:31.573231 22815 ts_tablet_manager.cc:1397] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:31.573228 22817 ts_tablet_manager.cc:1397] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:31.573458 22816 tablet_bootstrap.cc:654] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:31.574218 22816 log.cc:824] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:31.575135 22816 tablet_bootstrap.cc:492] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e: No bootstrap required, opened a new log
I20260620 23:01:31.575259 22816 ts_tablet_manager.cc:1397] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:01:31.575423 22817 raft_consensus.cc:348] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.575596 22817 raft_consensus.cc:374] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:31.575639 22817 raft_consensus.cc:729] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb0967483304340be28c3c654342eef, State: Initialized, Role: FOLLOWER
I20260620 23:01:31.575780 22817 consensus_queue.cc:260] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.575771 22815 raft_consensus.cc:348] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.575984 22815 raft_consensus.cc:374] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:31.576028 22815 raft_consensus.cc:729] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3da2defc2d466b8600c9f7050b2d1c, State: Initialized, Role: FOLLOWER
I20260620 23:01:31.576107 22817 ts_tablet_manager.cc:1428] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:31.576156 22815 consensus_queue.cc:260] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.576236 22664 heartbeater.cc:503] Master 127.21.66.126:34633 was elected leader, sending a full tablet report...
I20260620 23:01:31.576483 22530 heartbeater.cc:503] Master 127.21.66.126:34633 was elected leader, sending a full tablet report...
I20260620 23:01:31.576722 22815 ts_tablet_manager.cc:1428] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 23:01:31.577701 22816 raft_consensus.cc:348] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.577894 22816 raft_consensus.cc:374] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:31.577924 22816 raft_consensus.cc:729] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79979e523444e29a73446edf300238e, State: Initialized, Role: FOLLOWER
I20260620 23:01:31.578045 22816 consensus_queue.cc:260] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.578404 22816 ts_tablet_manager.cc:1428] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 23:01:31.578424 22798 heartbeater.cc:503] Master 127.21.66.126:34633 was elected leader, sending a full tablet report...
I20260620 23:01:31.578421 22822 raft_consensus.cc:482] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:31.578497 22822 raft_consensus.cc:504] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.578905 22822 leader_election.cc:288] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfb0967483304340be28c3c654342eef (127.21.66.66:46089), f79979e523444e29a73446edf300238e (127.21.66.67:35041)
I20260620 23:01:31.581880 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.579231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:34265 (local address 127.21.66.67:35041)
0620 23:01:31.579278 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.579280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.579288 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:31.579320 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.579321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.579330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.579390 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:31.579532 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.580000 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.581090 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.581469 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.581566 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.581575 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.581773 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.581808 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:01:31.581929 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.579033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:35041 (local address 127.21.66.65:34265)
0620 23:01:31.579179 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.579190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.579241 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.579395 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.579396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.579397 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:31.579470 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.579472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.580091 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.580093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.580996 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.580999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.581567 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.581568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.581678 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.581682 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.581764 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.581806 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":43,"thread_start_us":98,"threads_started":1}
I20260620 23:01:31.581931 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.579055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:48485 (local address 127.21.66.66:46089)
0620 23:01:31.579101 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.579103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.579113 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:31.579179 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.579181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.579191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.579245 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:31.579484 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.580036 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.581115 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.581519 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.581622 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.581632 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.581821 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.581859 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:31.581941 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.579014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:46089 (local address 127.21.66.65:48485)
0620 23:01:31.579066 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:31.579079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:31.579123 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:31.579301 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:31.579302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:31.579303 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:31.579383 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.579385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.580133 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.580134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.581009 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:31.581012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.581622 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.581623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:31.581718 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.581721 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:31.581805 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:01:31.581849 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":35}
I20260620 23:01:31.582332 22618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbac8b2eeb4bffa0257e5ac0b720dc" candidate_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb0967483304340be28c3c654342eef" is_pre_election: true
I20260620 23:01:31.582333 22752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbac8b2eeb4bffa0257e5ac0b720dc" candidate_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79979e523444e29a73446edf300238e" is_pre_election: true
I20260620 23:01:31.582469 22618 raft_consensus.cc:2454] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee3da2defc2d466b8600c9f7050b2d1c in term 0.
I20260620 23:01:31.582466 22752 raft_consensus.cc:2454] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee3da2defc2d466b8600c9f7050b2d1c in term 0.
I20260620 23:01:31.582679 22417 leader_election.cc:302] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee3da2defc2d466b8600c9f7050b2d1c, f79979e523444e29a73446edf300238e; no voters:
I20260620 23:01:31.582871 22822 raft_consensus.cc:2781] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:31.582940 22822 raft_consensus.cc:482] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:31.582985 22822 raft_consensus.cc:3037] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:31.583705 22822 raft_consensus.cc:504] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.583868 22822 leader_election.cc:288] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [CANDIDATE]: Term 1 election: Requested vote from peers cfb0967483304340be28c3c654342eef (127.21.66.66:46089), f79979e523444e29a73446edf300238e (127.21.66.67:35041)
I20260620 23:01:31.584022 22618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbac8b2eeb4bffa0257e5ac0b720dc" candidate_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb0967483304340be28c3c654342eef"
I20260620 23:01:31.584021 22752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02bbac8b2eeb4bffa0257e5ac0b720dc" candidate_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79979e523444e29a73446edf300238e"
I20260620 23:01:31.584105 22752 raft_consensus.cc:3037] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:31.584105 22618 raft_consensus.cc:3037] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:31.585052 22618 raft_consensus.cc:2454] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee3da2defc2d466b8600c9f7050b2d1c in term 1.
I20260620 23:01:31.585052 22752 raft_consensus.cc:2454] T 02bbac8b2eeb4bffa0257e5ac0b720dc P f79979e523444e29a73446edf300238e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee3da2defc2d466b8600c9f7050b2d1c in term 1.
I20260620 23:01:31.585219 22417 leader_election.cc:302] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee3da2defc2d466b8600c9f7050b2d1c, f79979e523444e29a73446edf300238e; no voters:
I20260620 23:01:31.585356 22822 raft_consensus.cc:2781] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:31.585577 22822 raft_consensus.cc:686] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [term 1 LEADER]: Becoming Leader. State: Replica: ee3da2defc2d466b8600c9f7050b2d1c, State: Running, Role: LEADER
I20260620 23:01:31.585695 22822 consensus_queue.cc:237] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } }
I20260620 23:01:31.586673 22338 catalog_manager.cc:5495] T 02bbac8b2eeb4bffa0257e5ac0b720dc P ee3da2defc2d466b8600c9f7050b2d1c reported cstate change: term changed from 0 to 1, leader changed from <none> to ee3da2defc2d466b8600c9f7050b2d1c (127.21.66.65). New cstate: current_term: 1 leader_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfb0967483304340be28c3c654342eef" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f79979e523444e29a73446edf300238e" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee3da2defc2d466b8600c9f7050b2d1c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42879 } health_report { overall_health: HEALTHY } } }
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
W20260620 23:01:31.594594 22665 tablet.cc:2369] T 02bbac8b2eeb4bffa0257e5ac0b720dc P cfb0967483304340be28c3c654342eef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:01:31.598062 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.590247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.21.66.65:42879)
0620 23:01:31.590312 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.590314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.590439 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:01:31.590519 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.590521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.590532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.590591 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:31.591173 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.591674 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.592416 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.592609 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.592693 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.595080 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:31.595089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:31.595090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:31.595097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:31.596764 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.597116 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.597118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.597119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.597173 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.597407 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.597408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.597408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.597550 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:31.597655 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.597829 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.597894 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:31.622792 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.614843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.21.66.65:42879)
0620 23:01:31.614932 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.614934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.614993 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:01:31.615047 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.615048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.615056 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.615109 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:31.615677 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.616257 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.617087 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.617294 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.617363 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.619950 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:31.619952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:31.619952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:31.619959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:31.621393 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.621742 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.621743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.621743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.621809 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.621990 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.621991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.621991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.622092 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:31.622219 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.622477 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.622558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:31.648376 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:31.640304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.21.66.65:42879)
0620 23:01:31.640382 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:31.640384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:31.640552 (+ 168us) server_negotiation.cc:407] Connection header received
0620 23:01:31.640585 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:31.640586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:31.640597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:31.640659 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:31.641272 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.641833 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.642734 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:31.642952 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:31.643009 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:31.645529 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:31.645531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:31.645531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:31.645538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:31.647003 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.647403 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.647404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.647404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.647458 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:31.647749 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:31.647750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:31.647750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:31.647877 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:31.647988 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:31.648150 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:01:31.648224 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 23:01:31.648916 22444 tablet_service.cc:2908] Rejecting scan request for tablet 02bbac8b2eeb4bffa0257e5ac0b720dc: Uninitialized: clean time has not yet been initialized
I20260620 23:01:31.649899 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22401
I20260620 23:01:31.655123 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22535
I20260620 23:01:31.660944 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22669
I20260620 23:01:31.665519 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22307
2026-06-20T23:01:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2060 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22839](info): set up 2 sockets
Jun 20 23:01:31 dist-test-slave-qgtk krb5kdc[22839](info): commencing operation
krb5kdc: starting...
W20260620 23:01:32.742246 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:32 dist-test-slave-qgtk krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:32.873167 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:41509
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:41509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:32.964378 22852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:32.964679 22852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:32.964728 22852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:32.966202 22852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:32.966291 22852 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:01:32.966315 22852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:32.966341 22852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:32.966398 22852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:32.966431 22852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:32.968207 22852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:41509
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:41509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:32.968581 22852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:32.968859 22852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:32.974447 22857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:32.974412 22858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:32.974408 22860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:32.974936 22852 server_base.cc:1034] running on GCE node
I20260620 23:01:32.975219 22852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:32.975556 22852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:32.976764 22852 hybrid_clock.cc:648] HybridClock initialized: now 1781996492976749 us; error 51 us; skew 500 ppm
Jun 20 23:01:32 dist-test-slave-qgtk krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996492, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:32.983350 22852 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:32.983937 22852 webserver.cc:460] Webserver started at http://127.21.66.126:34243/ using document root <none> and password file <none>
I20260620 23:01:32.984189 22852 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:32.984259 22852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:32.984412 22852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:32.985550 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "cf6d93bc0df548479ae0f8b3ae5225e4"
format_stamp: "Formatted at 2026-06-20 23:01:32 on dist-test-slave-qgtk"
server_key: "aa3bdb977a70a78ad0ab9687e95a4566"
server_key_iv: "62a1f24655f477dae3acae0833e57a3b"
server_key_version: "encryptionkey@0"
I20260620 23:01:32.985972 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "cf6d93bc0df548479ae0f8b3ae5225e4"
format_stamp: "Formatted at 2026-06-20 23:01:32 on dist-test-slave-qgtk"
server_key: "aa3bdb977a70a78ad0ab9687e95a4566"
server_key_iv: "62a1f24655f477dae3acae0833e57a3b"
server_key_version: "encryptionkey@0"
I20260620 23:01:32.987510 22852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:32.988505 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:32.988770 22852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:32.988909 22852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "cf6d93bc0df548479ae0f8b3ae5225e4"
format_stamp: "Formatted at 2026-06-20 23:01:32 on dist-test-slave-qgtk"
server_key: "aa3bdb977a70a78ad0ab9687e95a4566"
server_key_iv: "62a1f24655f477dae3acae0833e57a3b"
server_key_version: "encryptionkey@0"
I20260620 23:01:32.989030 22852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:33.040792 22852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:33.043463 22852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:33.043655 22852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:33.048484 22852 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:41509
I20260620 23:01:33.048643 22929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:41509 every 8 connection(s)
I20260620 23:01:33.048941 22852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:33.049480 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22852
I20260620 23:01:33.049592 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:33.049763 21769 external_mini_cluster.cc:1428] Setting key 8011f1bd505a8da0fa81bcadc3706f4c
I20260620 23:01:33.049793 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:33.053474 22930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4: Bootstrap starting.
I20260620 23:01:33.054332 22930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:33.054685 22930 log.cc:824] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:33.055523 22930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4: No bootstrap required, opened a new log
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:33.058086 22930 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 41509 } }
I20260620 23:01:33.058336 22930 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:33.058367 22930 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6d93bc0df548479ae0f8b3ae5225e4, State: Initialized, Role: FOLLOWER
I20260620 23:01:33.058509 22930 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 41509 } }
I20260620 23:01:33.058605 22930 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:33.058642 22930 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:33.058692 22930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:33.059445 22930 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 41509 } }
I20260620 23:01:33.059545 22930 leader_election.cc:302] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf6d93bc0df548479ae0f8b3ae5225e4; no voters:
I20260620 23:01:33.059805 22930 leader_election.cc:288] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:33.059854 22935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:33.060029 22935 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [term 1 LEADER]: Becoming Leader. State: Replica: cf6d93bc0df548479ae0f8b3ae5225e4, State: Running, Role: LEADER
I20260620 23:01:33.060133 22930 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:33.060205 22935 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 41509 } }
I20260620 23:01:33.060606 22935 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf6d93bc0df548479ae0f8b3ae5225e4. Latest consensus state: current_term: 1 leader_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 41509 } } }
I20260620 23:01:33.060791 22935 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:33.061293 22944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:33.061614 22937 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf6d93bc0df548479ae0f8b3ae5225e4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 41509 } } }
I20260620 23:01:33.061771 22937 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:33.061965 22944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:33.062634 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.050362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.21.66.126:41509)
0620 23:01:33.050531 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.050540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.050575 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:33.051033 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.051040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.051087 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.051219 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.052171 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.053142 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.054339 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.054887 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.055033 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.058106 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.058110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.058110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.058166 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.060384 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.060851 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.060854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.060855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.060982 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.061207 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.061207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.061208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.061546 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.061705 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.062273 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.062475 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260620 23:01:33.063730 22944 catalog_manager.cc:1347] Generated new cluster ID: 98b3356162104274945ff8f2b0e6d73f
I20260620 23:01:33.063783 22944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:33.080111 22944 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:33.080892 22944 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:33.088224 22944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4: Generated new TSK 0
I20260620 23:01:33.088452 22944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:33.121893 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:41509
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:33.236620 22956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:33.236862 22956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:33.236907 22956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:33.238197 22956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:33.238260 22956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:33.238297 22956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:01:33.238336 22956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:33.239804 22956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:41509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:33.240075 22956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:33.240315 22956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:33.245457 22961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:33.245623 22956 server_base.cc:1034] running on GCE node
W20260620 23:01:33.245582 22962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:33.245471 22964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:33.246021 22956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:33.246336 22956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:33.247597 22956 hybrid_clock.cc:648] HybridClock initialized: now 1781996493247574 us; error 31 us; skew 500 ppm
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:33.250600 22956 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:33.251106 22956 webserver.cc:460] Webserver started at http://127.21.66.65:35365/ using document root <none> and password file <none>
I20260620 23:01:33.251371 22956 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:33.251422 22956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:33.251616 22956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:33.252817 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "8748ab320aaa4548ba51cea73d5f5e69"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "85e0451d15688c87cb0ee26e461f4400"
server_key_iv: "747dd6267ebee5a6cdeacf25077f3456"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.253232 22956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "8748ab320aaa4548ba51cea73d5f5e69"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "85e0451d15688c87cb0ee26e461f4400"
server_key_iv: "747dd6267ebee5a6cdeacf25077f3456"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.254890 22956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:33.255805 22971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:33.256065 22956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:33.256196 22956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "8748ab320aaa4548ba51cea73d5f5e69"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "85e0451d15688c87cb0ee26e461f4400"
server_key_iv: "747dd6267ebee5a6cdeacf25077f3456"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.256289 22956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:33.271382 22956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:33.274302 22956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:33.274509 22956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:33.275164 22956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:33.275236 22956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:33.275286 22956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:33.275318 22956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:33.282473 22956 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:42317
I20260620 23:01:33.282614 23086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:42317 every 8 connection(s)
I20260620 23:01:33.282950 22956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:33.288138 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.275836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:42583 (local address 127.21.66.126:41509)
0620 23:01:33.275966 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.275969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.276329 (+ 360us) server_negotiation.cc:407] Connection header received
0620 23:01:33.277205 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.277207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.277223 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.277325 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.278382 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.279324 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.280936 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.281348 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.281433 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.284872 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.284874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.284874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.284882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.286535 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.287065 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.287066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.287067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.287135 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.287479 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.287480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.287481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.287627 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.287700 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.287985 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.288069 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:01:33.288347 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 22956
I20260620 23:01:33.288393 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.275987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:41509 (local address 127.21.66.65:42583)
0620 23:01:33.276190 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.276237 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.277034 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.277418 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.277422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.277867 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:33.278233 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.278242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.279435 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.279436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.280840 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.280845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.281460 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.281461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.281669 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.282508 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:33.282521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:33.284702 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:33.286662 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.286670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.286672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.286948 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.287238 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.287239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.287240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.287369 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.287703 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:33.287709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:33.287838 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.287929 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.288070 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":119,"threads_started":1}
I20260620 23:01:33.288456 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:33.288631 21769 external_mini_cluster.cc:1428] Setting key afca6f373f42a6ade124c8446c356e2a
I20260620 23:01:33.289013 23087 heartbeater.cc:347] Connected to a master server at 127.21.66.126:41509
I20260620 23:01:33.289156 23087 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:33.289392 23087 heartbeater.cc:511] Master 127.21.66.126:41509 requested a full tablet report, sending...
I20260620 23:01:33.291424 22883 ts_manager.cc:194] Registered new tserver with Master: 8748ab320aaa4548ba51cea73d5f5e69 (127.21.66.65:42317)
I20260620 23:01:33.292199 22883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:42583
I20260620 23:01:33.297206 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.290069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:41509 (local address 127.21.66.65:39831)
0620 23:01:33.290266 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.290276 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.290356 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.290504 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.290505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.290749 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:33.290848 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.290850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.291781 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.291782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.292606 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.292609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.293158 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.293159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.293235 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.293766 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:33.293771 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:33.294251 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:33.296172 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.296173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.296174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.296354 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.296595 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.296596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.296596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.296656 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.296942 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:33.296943 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:33.296983 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.297034 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.297126 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
I20260620 23:01:33.297266 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.290161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:39831 (local address 127.21.66.126:41509)
0620 23:01:33.290205 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.290208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.290341 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:01:33.290415 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.290416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.290427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.290499 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.290947 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.291678 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.292714 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.293078 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.293151 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.294342 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.294343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.294344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.294351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.296081 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.296432 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.296434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.296435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.296519 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.296721 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.296722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.296722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.296876 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.296948 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.297064 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.297153 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:33.321499 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:41509
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:33.406814 23096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:33.407038 23096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:33.407065 23096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:33.408390 23096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:33.408450 23096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:33.408473 23096 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:01:33.408492 23096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:33.410086 23096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:41509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:33.410432 23096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:33.410710 23096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:33.416254 23104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:33.416421 23102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:33.416254 23101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:33.416517 23096 server_base.cc:1034] running on GCE node
I20260620 23:01:33.416765 23096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:33.417065 23096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:33.418280 23096 hybrid_clock.cc:648] HybridClock initialized: now 1781996493418265 us; error 32 us; skew 500 ppm
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:33.421130 23096 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:33.421752 23096 webserver.cc:460] Webserver started at http://127.21.66.66:46365/ using document root <none> and password file <none>
I20260620 23:01:33.422040 23096 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:33.422114 23096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:33.422288 23096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:33.423444 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "845bd6451bb24cf493281fab5cd0109e"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "1c17ebd23eb033dbd4ab7387750d27e9"
server_key_iv: "584e0755f14ef0524006db68533b2fbb"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.423856 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "845bd6451bb24cf493281fab5cd0109e"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "1c17ebd23eb033dbd4ab7387750d27e9"
server_key_iv: "584e0755f14ef0524006db68533b2fbb"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.425410 23096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:33.426522 23111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:33.426972 23096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:33.427109 23096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "845bd6451bb24cf493281fab5cd0109e"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "1c17ebd23eb033dbd4ab7387750d27e9"
server_key_iv: "584e0755f14ef0524006db68533b2fbb"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.427204 23096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:33.444121 23096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:33.446803 23096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:33.446991 23096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:33.447738 23096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:33.447799 23096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:33.447837 23096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:33.447870 23096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:33.455567 23096 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:45909
I20260620 23:01:33.455765 23226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:45909 every 8 connection(s)
I20260620 23:01:33.456044 23096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:01:33.457487 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 23096
I20260620 23:01:33.457594 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:33.457798 21769 external_mini_cluster.cc:1428] Setting key 363dc1f8149a19f1fe8159ad5f270dc3
I20260620 23:01:33.459092 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.448259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:55079 (local address 127.21.66.126:41509)
0620 23:01:33.448334 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.448337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.448577 (+ 240us) server_negotiation.cc:407] Connection header received
0620 23:01:33.449103 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.449105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.449118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.449198 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.450232 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.451089 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.452313 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.452677 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.452769 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.455413 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.455415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.455415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.455427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.457139 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.457622 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.457623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.457623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.457689 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.458246 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.458247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.458248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.458404 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.458515 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.458879 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.459012 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:01:33.459323 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.448232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:41509 (local address 127.21.66.66:55079)
0620 23:01:33.448428 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.448476 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.448998 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.449326 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.449331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.449722 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:33.450077 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.450084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.451222 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.451224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.452185 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.452189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.452797 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.452799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.452938 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.453553 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:33.453562 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:33.455274 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:33.457225 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.457232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.457233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.457491 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.457891 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.457892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.457892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.458085 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.458501 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:33.458506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:33.458700 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.458821 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.459017 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260620 23:01:33.459928 23227 heartbeater.cc:347] Connected to a master server at 127.21.66.126:41509
I20260620 23:01:33.460072 23227 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:33.460325 23227 heartbeater.cc:511] Master 127.21.66.126:41509 requested a full tablet report, sending...
I20260620 23:01:33.460736 22883 ts_manager.cc:194] Registered new tserver with Master: 845bd6451bb24cf493281fab5cd0109e (127.21.66.66:45909)
I20260620 23:01:33.461272 22883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:55079
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
I20260620 23:01:33.473161 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.461034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:41509 (local address 127.21.66.66:59693)
0620 23:01:33.466357 (+ 5323us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.466373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.466465 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.466669 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.466670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.466880 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:33.467006 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.467009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.467860 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.467861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.468533 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.468536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.468929 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.468930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.468997 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.469552 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:33.469557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:33.469910 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:33.471810 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.471811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.471812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.471999 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.472310 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.472311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.472312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.472379 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.472800 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:33.472801 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:33.472859 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.472939 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.473051 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5283,"thread_start_us":69,"threads_started":1}
I20260620 23:01:33.473227 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.461100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:59693 (local address 127.21.66.126:41509)
0620 23:01:33.461270 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.461272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.466496 (+ 5224us) server_negotiation.cc:407] Connection header received
0620 23:01:33.466543 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.466545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.466564 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.466652 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.467099 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.467733 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.468638 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.468836 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.468912 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.470030 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.470031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.470032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.470040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.471710 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.472074 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.472075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.472076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.472192 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.472452 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.472453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.472454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.472689 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.472788 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.472988 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.473074 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:33.489887 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:41509
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:33.575700 23236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:33.576061 23236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:33.576093 23236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:33.577342 23236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:33.577390 23236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:33.577412 23236 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:01:33.577430 23236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:33.579010 23236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:41509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:33.579268 23236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:33.579468 23236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:33.584885 23244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:33.584844 23242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:33.584844 23241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:33.585382 23236 server_base.cc:1034] running on GCE node
I20260620 23:01:33.585595 23236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:33.585924 23236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:33.587131 23236 hybrid_clock.cc:648] HybridClock initialized: now 1781996493587078 us; error 67 us; skew 500 ppm
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:33.590216 23236 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:33.590719 23236 webserver.cc:460] Webserver started at http://127.21.66.67:34143/ using document root <none> and password file <none>
I20260620 23:01:33.590962 23236 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:33.591008 23236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:33.591156 23236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:33.592439 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "57f53ee0561045969e4af5f80d088796"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "73452dd0510e796ef1a4e2ccd660814a"
server_key_iv: "036fed695cdbfa36dc57a629d3c8b29d"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.592855 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "57f53ee0561045969e4af5f80d088796"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "73452dd0510e796ef1a4e2ccd660814a"
server_key_iv: "036fed695cdbfa36dc57a629d3c8b29d"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.594328 23236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:33.595240 23251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:33.595433 23236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:33.595537 23236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "57f53ee0561045969e4af5f80d088796"
format_stamp: "Formatted at 2026-06-20 23:01:33 on dist-test-slave-qgtk"
server_key: "73452dd0510e796ef1a4e2ccd660814a"
server_key_iv: "036fed695cdbfa36dc57a629d3c8b29d"
server_key_version: "encryptionkey@0"
I20260620 23:01:33.595608 23236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:33.623896 23236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:33.626888 23236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:33.627023 23236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:33.627707 23236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:33.627768 23236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:33.627806 23236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:33.627832 23236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:33.636461 23236 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:35257
I20260620 23:01:33.636548 23366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:35257 every 8 connection(s)
I20260620 23:01:33.636894 23236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:01:33.639925 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.628289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:41787 (local address 127.21.66.126:41509)
0620 23:01:33.628367 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.628370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.628664 (+ 294us) server_negotiation.cc:407] Connection header received
0620 23:01:33.629500 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.629504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.629522 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.629608 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.630720 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.631423 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.633381 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.633723 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.633814 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.636812 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.636813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.636814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.636823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.638410 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.638939 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.638940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.638941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.638995 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.639258 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.639259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.639259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.639370 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.639423 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.639790 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.639857 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:01:33.640221 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.628249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:41509 (local address 127.21.66.67:41787)
0620 23:01:33.628486 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.628545 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.629355 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.629714 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.629719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.630218 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:33.630601 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.630610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.631535 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.631537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.633283 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.633287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.633858 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.633859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.634052 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.634660 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:33.634681 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:33.636680 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:33.638492 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.638499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.638501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.638862 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.639057 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.639058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.639058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.639174 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.639455 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:33.639461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:33.639605 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.639766 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.639906 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
I20260620 23:01:33.640810 23367 heartbeater.cc:347] Connected to a master server at 127.21.66.126:41509
I20260620 23:01:33.640915 23367 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:33.641098 23367 heartbeater.cc:511] Master 127.21.66.126:41509 requested a full tablet report, sending...
I20260620 23:01:33.641572 22884 ts_manager.cc:194] Registered new tserver with Master: 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257)
I20260620 23:01:33.642059 22884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:41787
I20260620 23:01:33.646037 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 23236
I20260620 23:01:33.646189 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:33.646397 21769 external_mini_cluster.cc:1428] Setting key 596f07fa7b245344db8ec8e6fc4aab60
I20260620 23:01:33.647467 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:01:33.651062 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.641964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:34415 (local address 127.21.66.126:41509)
0620 23:01:33.642005 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.642007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.642228 (+ 221us) server_negotiation.cc:407] Connection header received
0620 23:01:33.642293 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.642294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.642304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.642366 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.642704 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.643182 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.643920 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.644120 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.644205 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.645200 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.645201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.645202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.645208 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.649649 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.650113 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.650114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.650114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.650251 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.650512 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.650513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.650513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.650640 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.650706 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.650924 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.650997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:01:33.651160 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.641875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:41509 (local address 127.21.66.67:34415)
0620 23:01:33.642168 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.642180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.642240 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.642370 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.642370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.642534 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:33.642619 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.642621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.643278 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.643279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.643802 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.643803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.644205 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.644206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.644280 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.644786 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:33.644792 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:33.645085 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:33.649759 (+ 4674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.649760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.649761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.650022 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.650327 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:33.650328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:33.650328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:33.650411 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:33.650716 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:33.650717 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:33.650779 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.650854 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.650989 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":6,"thread_start_us":80,"threads_started":1}
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:01:33 dist-test-slave-qgtk krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:33.675340 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.665747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.21.66.126:41509)
0620 23:01:33.665830 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.665832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.665842 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:33.665925 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.665926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.665934 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.665987 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:33.667076 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.667905 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.669242 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.669532 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.669617 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.672365 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:33.672367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:33.672367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:33.672376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:33.673988 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.674421 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.674422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.674423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.674477 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:33.674639 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:33.674640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:33.674640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:33.674747 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:33.674821 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.675056 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.675130 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:01:33.677392 22884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:33.679205 22884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:33.686354 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.682277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:45909 (local address 127.0.0.1:59344)
0620 23:01:33.682383 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.682392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.682434 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.682746 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.682749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.682751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:33.682858 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.682868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.683803 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.683804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.685031 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.685033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.685991 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.685992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.686080 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.686096 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.686210 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.686268 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260620 23:01:33.686486 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.681961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:35257 (local address 127.0.0.1:52028)
0620 23:01:33.682075 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.682116 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.682208 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.682752 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.682753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.682756 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:33.682869 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.682872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.684220 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.684221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.685218 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.685220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.686163 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.686164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.686251 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.686255 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.686341 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.686401 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":20,"thread_start_us":66,"threads_started":1}
I20260620 23:01:33.686554 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.682423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.21.66.67:35257)
0620 23:01:33.682567 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.682572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.682584 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:01:33.682625 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.682627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.682642 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.682941 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:33.683000 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.684135 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.685316 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.686060 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.686162 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.686224 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.686361 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.686411 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260620 23:01:33.686765 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.682393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.21.66.66:45909)
0620 23:01:33.682558 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.682563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.682573 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:33.682616 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.682618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.682635 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.682946 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:33.683007 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.683711 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.685117 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.685895 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.685977 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.686041 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.686602 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.686659 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260620 23:01:33.687470 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.682045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42317 (local address 127.0.0.1:36850)
0620 23:01:33.682360 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.682371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.682423 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.682925 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.682925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.682928 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:33.682971 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.682972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.685305 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.685306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.685999 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.686000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.687263 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.687264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.687337 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.687342 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.687373 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.687414 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":114,"threads_started":1}
I20260620 23:01:33.687462 23301 tablet_service.cc:1467] Processing CreateTablet for tablet 08f2a314c4524df796b7c612bf6e180e (DEFAULT_TABLE table=test-table [id=d22c2640f1804ddc881c41fbd1153a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:33.687462 23161 tablet_service.cc:1467] Processing CreateTablet for tablet 08f2a314c4524df796b7c612bf6e180e (DEFAULT_TABLE table=test-table [id=d22c2640f1804ddc881c41fbd1153a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:33.687831 23301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f2a314c4524df796b7c612bf6e180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:33.687831 23161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f2a314c4524df796b7c612bf6e180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:33.687844 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.682168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.21.66.65:42317)
0620 23:01:33.682648 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.682652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.682747 (+ 95us) server_negotiation.cc:407] Connection header received
0620 23:01:33.682791 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.682794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.682809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.682927 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:33.683037 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.684122 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.686092 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.687158 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.687598 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.687673 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.687699 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.687739 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":86,"threads_started":1}
I20260620 23:01:33.691473 23390 tablet_bootstrap.cc:492] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Bootstrap starting.
I20260620 23:01:33.692049 23021 tablet_service.cc:1467] Processing CreateTablet for tablet 08f2a314c4524df796b7c612bf6e180e (DEFAULT_TABLE table=test-table [id=d22c2640f1804ddc881c41fbd1153a73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:33.692381 23021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f2a314c4524df796b7c612bf6e180e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:33.692991 23390 tablet_bootstrap.cc:654] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:33.693346 23390 log.cc:824] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:33.694310 23391 tablet_bootstrap.cc:492] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Bootstrap starting.
I20260620 23:01:33.694466 23390 tablet_bootstrap.cc:492] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: No bootstrap required, opened a new log
I20260620 23:01:33.694548 23390 ts_tablet_manager.cc:1397] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:01:33.694988 23391 tablet_bootstrap.cc:654] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:33.695415 23391 log.cc:824] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:33.696180 23394 tablet_bootstrap.cc:492] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Bootstrap starting.
I20260620 23:01:33.696280 23391 tablet_bootstrap.cc:492] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: No bootstrap required, opened a new log
I20260620 23:01:33.696357 23391 ts_tablet_manager.cc:1397] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:33.696753 23390 raft_consensus.cc:348] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.696956 23390 raft_consensus.cc:374] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:33.697005 23390 raft_consensus.cc:729] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f53ee0561045969e4af5f80d088796, State: Initialized, Role: FOLLOWER
I20260620 23:01:33.697027 23394 tablet_bootstrap.cc:654] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:33.697197 23390 consensus_queue.cc:260] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.697400 23394 log.cc:824] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:33.697526 23390 ts_tablet_manager.cc:1428] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:01:33.697678 23367 heartbeater.cc:503] Master 127.21.66.126:41509 was elected leader, sending a full tablet report...
I20260620 23:01:33.698372 23394 tablet_bootstrap.cc:492] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: No bootstrap required, opened a new log
I20260620 23:01:33.698335 23391 raft_consensus.cc:348] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.698477 23394 ts_tablet_manager.cc:1397] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:33.698499 23391 raft_consensus.cc:374] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:33.698540 23391 raft_consensus.cc:729] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845bd6451bb24cf493281fab5cd0109e, State: Initialized, Role: FOLLOWER
I20260620 23:01:33.698709 23391 consensus_queue.cc:260] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.699010 23391 ts_tablet_manager.cc:1428] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:33.699128 23227 heartbeater.cc:503] Master 127.21.66.126:41509 was elected leader, sending a full tablet report...
I20260620 23:01:33.700429 23394 raft_consensus.cc:348] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.700588 23394 raft_consensus.cc:374] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:33.700639 23394 raft_consensus.cc:729] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8748ab320aaa4548ba51cea73d5f5e69, State: Initialized, Role: FOLLOWER
I20260620 23:01:33.700780 23394 consensus_queue.cc:260] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.701042 23394 ts_tablet_manager.cc:1428] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:33.701124 23087 heartbeater.cc:503] Master 127.21.66.126:41509 was elected leader, sending a full tablet report...
W20260620 23:01:33.706717 23228 tablet.cc:2369] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:01:33.783898 23088 tablet.cc:2369] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:01:33.887746 23368 tablet.cc:2369] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:01:33.971979 23398 raft_consensus.cc:482] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:33.972160 23398 raft_consensus.cc:504] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.972589 23398 leader_election.cc:288] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 845bd6451bb24cf493281fab5cd0109e (127.21.66.66:45909), 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257)
I20260620 23:01:33.975795 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.972917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:49071 (local address 127.21.66.67:35257)
0620 23:01:33.972995 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.972997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.973009 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:01:33.973091 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.973092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.973102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.973169 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:33.973367 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.973965 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.975045 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.975426 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.975512 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.975521 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.975682 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.975721 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:33.975823 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.972800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:35257 (local address 127.21.66.65:49071)
0620 23:01:33.972930 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.972948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.973024 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.973175 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.973176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.973179 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:33.973260 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.973261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.974090 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.974092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.974948 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.974950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.975521 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.975522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.975599 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.975603 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.975676 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.975720 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260620 23:01:33.975859 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.973101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:45413 (local address 127.21.66.66:45909)
0620 23:01:33.973148 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:33.973150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:33.973246 (+ 96us) server_negotiation.cc:407] Connection header received
0620 23:01:33.973317 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:33.973318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:33.973326 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:33.973403 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:33.973581 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.974052 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.975045 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.975500 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.975584 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.975592 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:33.975750 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:01:33.975789 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:33.976089 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:33.973034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:45909 (local address 127.21.66.65:45413)
0620 23:01:33.973168 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:33.973178 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:33.973255 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:33.973406 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:33.973406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:33.973409 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:33.973470 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.973472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.974163 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.974164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.974950 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:33.974952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:33.975583 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:33.975584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:33.975658 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:33.975667 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:01:33.975742 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:01:33.975786 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260620 23:01:33.976169 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f2a314c4524df796b7c612bf6e180e" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f53ee0561045969e4af5f80d088796" is_pre_election: true
I20260620 23:01:33.976341 23321 raft_consensus.cc:2454] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 0.
I20260620 23:01:33.976387 23181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f2a314c4524df796b7c612bf6e180e" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845bd6451bb24cf493281fab5cd0109e" is_pre_election: true
I20260620 23:01:33.976538 23181 raft_consensus.cc:2454] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 0.
I20260620 23:01:33.976596 22973 leader_election.cc:302] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f53ee0561045969e4af5f80d088796, 8748ab320aaa4548ba51cea73d5f5e69; no voters:
I20260620 23:01:33.976725 23398 raft_consensus.cc:2781] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:33.976791 23398 raft_consensus.cc:482] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:33.976809 23398 raft_consensus.cc:3037] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:33.977568 23398 raft_consensus.cc:504] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.977715 23398 leader_election.cc:288] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 election: Requested vote from peers 845bd6451bb24cf493281fab5cd0109e (127.21.66.66:45909), 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257)
I20260620 23:01:33.977924 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f2a314c4524df796b7c612bf6e180e" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f53ee0561045969e4af5f80d088796"
I20260620 23:01:33.977914 23181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f2a314c4524df796b7c612bf6e180e" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845bd6451bb24cf493281fab5cd0109e"
I20260620 23:01:33.977996 23321 raft_consensus.cc:3037] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:33.978035 23181 raft_consensus.cc:3037] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:33.978701 23321 raft_consensus.cc:2454] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 1.
I20260620 23:01:33.978870 22973 leader_election.cc:302] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f53ee0561045969e4af5f80d088796, 8748ab320aaa4548ba51cea73d5f5e69; no voters:
I20260620 23:01:33.978930 23181 raft_consensus.cc:2454] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 1.
I20260620 23:01:33.978956 23398 raft_consensus.cc:2781] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:33.979063 23398 raft_consensus.cc:686] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 LEADER]: Becoming Leader. State: Replica: 8748ab320aaa4548ba51cea73d5f5e69, State: Running, Role: LEADER
I20260620 23:01:33.979156 23398 consensus_queue.cc:237] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:33.980219 22883 catalog_manager.cc:5495] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8748ab320aaa4548ba51cea73d5f5e69 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "8748ab320aaa4548ba51cea73d5f5e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:34.042578 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.039878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.21.66.65:42317)
0620 23:01:34.039955 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.039957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.039967 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:34.040052 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.040053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.040079 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.040155 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:34.040298 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.040803 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.041672 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.041871 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.041964 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.042205 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:34.042310 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:34.042403 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.042462 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.042487 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:01:34.044514 23321 raft_consensus.cc:1261] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Refusing update from remote peer 8748ab320aaa4548ba51cea73d5f5e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:34.044523 23181 raft_consensus.cc:1261] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Refusing update from remote peer 8748ab320aaa4548ba51cea73d5f5e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:34.044881 23398 consensus_queue.cc:1035] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:34.045091 23401 consensus_queue.cc:1035] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:34.048864 23406 mvcc.cc:205] Tried to move back new op lower bound from 7299057639602692096 to 7299057639338860544. Current Snapshot: MvccSnapshot[applied={T|T < 7299057639602692096}]
I20260620 23:01:34.050478 23405 mvcc.cc:205] Tried to move back new op lower bound from 7299057639602692096 to 7299057639338860544. Current Snapshot: MvccSnapshot[applied={T|T < 7299057639602692096 or (T in {7299057639602692096})}]
I20260620 23:01:34.053654 22884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:34.053774 22884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:34.054695 22884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4: Sending DeleteTablet for 3 replicas of tablet 08f2a314c4524df796b7c612bf6e180e
I20260620 23:01:34.055059 23021 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f2a314c4524df796b7c612bf6e180e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:34 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:36850
I20260620 23:01:34.055078 23301 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f2a314c4524df796b7c612bf6e180e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:34 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52028
I20260620 23:01:34.055114 23161 tablet_service.cc:1514] Processing DeleteTablet for tablet 08f2a314c4524df796b7c612bf6e180e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:34 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:59344
I20260620 23:01:34.055723 23415 tablet_replica.cc:330] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: stopping tablet replica
I20260620 23:01:34.055866 23415 raft_consensus.cc:2229] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:34.055796 22884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:34.055975 22884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:34.056035 23415 raft_consensus.cc:2258] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:34.056185 23413 tablet_replica.cc:330] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: stopping tablet replica
I20260620 23:01:34.056295 23413 raft_consensus.cc:2229] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:34.056406 23413 raft_consensus.cc:2258] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:34.056943 23413 ts_tablet_manager.cc:1905] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:34.057287 23161 tablet_service.cc:1467] Processing CreateTablet for tablet f449c9036a4247f1877a660950170d32 (DEFAULT_TABLE table=test-table [id=359602d393834dfd9cb5fd04ba142fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:34.057318 23301 tablet_service.cc:1467] Processing CreateTablet for tablet f449c9036a4247f1877a660950170d32 (DEFAULT_TABLE table=test-table [id=359602d393834dfd9cb5fd04ba142fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:34.057422 23161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f449c9036a4247f1877a660950170d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:34.057440 23301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f449c9036a4247f1877a660950170d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:34.058840 23390 tablet_bootstrap.cc:492] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: Bootstrap starting.
I20260620 23:01:34.058964 23391 tablet_bootstrap.cc:492] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: Bootstrap starting.
I20260620 23:01:34.059232 23021 tablet_service.cc:1467] Processing CreateTablet for tablet f449c9036a4247f1877a660950170d32 (DEFAULT_TABLE table=test-table [id=359602d393834dfd9cb5fd04ba142fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:34.059309 23390 tablet_bootstrap.cc:654] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:34.059341 23021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f449c9036a4247f1877a660950170d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:34.059414 23391 tablet_bootstrap.cc:654] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:34.059482 23415 ts_tablet_manager.cc:1905] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:34.059845 23413 ts_tablet_manager.cc:1918] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:34.060087 23413 log.cc:1196] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/08f2a314c4524df796b7c612bf6e180e
I20260620 23:01:34.060609 23413 ts_tablet_manager.cc:1939] T 08f2a314c4524df796b7c612bf6e180e P 845bd6451bb24cf493281fab5cd0109e: Deleting consensus metadata
I20260620 23:01:34.060801 23394 tablet_bootstrap.cc:492] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: Bootstrap starting.
I20260620 23:01:34.061272 23394 tablet_bootstrap.cc:654] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:34.061420 23415 ts_tablet_manager.cc:1918] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:34.061493 23415 log.cc:1196] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/08f2a314c4524df796b7c612bf6e180e
I20260620 23:01:34.061909 23414 tablet_replica.cc:330] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: stopping tablet replica
I20260620 23:01:34.062039 23394 tablet_bootstrap.cc:492] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: No bootstrap required, opened a new log
I20260620 23:01:34.061774 23415 ts_tablet_manager.cc:1939] T 08f2a314c4524df796b7c612bf6e180e P 57f53ee0561045969e4af5f80d088796: Deleting consensus metadata
I20260620 23:01:34.062096 23394 ts_tablet_manager.cc:1397] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:34.062271 23394 raft_consensus.cc:348] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.062371 23394 raft_consensus.cc:374] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:34.062393 23394 raft_consensus.cc:729] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8748ab320aaa4548ba51cea73d5f5e69, State: Initialized, Role: FOLLOWER
I20260620 23:01:34.062438 23394 consensus_queue.cc:260] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.062407 22870 catalog_manager.cc:4841] TS 845bd6451bb24cf493281fab5cd0109e (127.21.66.66:45909): tablet 08f2a314c4524df796b7c612bf6e180e (table test-table [id=d22c2640f1804ddc881c41fbd1153a73]) successfully deleted
I20260620 23:01:34.062512 23394 ts_tablet_manager.cc:1428] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:34.062541 22869 catalog_manager.cc:4841] TS 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257): tablet 08f2a314c4524df796b7c612bf6e180e (table test-table [id=d22c2640f1804ddc881c41fbd1153a73]) successfully deleted
I20260620 23:01:34.062644 23414 raft_consensus.cc:2229] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:34.062784 23414 raft_consensus.cc:2258] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:34.063318 23414 ts_tablet_manager.cc:1905] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:34.064927 23414 ts_tablet_manager.cc:1918] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:34.065011 23414 log.cc:1196] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/08f2a314c4524df796b7c612bf6e180e
I20260620 23:01:34.065107 23390 tablet_bootstrap.cc:492] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: No bootstrap required, opened a new log
I20260620 23:01:34.065166 23390 ts_tablet_manager.cc:1397] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 23:01:34.065346 23390 raft_consensus.cc:348] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.065402 23414 ts_tablet_manager.cc:1939] T 08f2a314c4524df796b7c612bf6e180e P 8748ab320aaa4548ba51cea73d5f5e69: Deleting consensus metadata
I20260620 23:01:34.065440 23390 raft_consensus.cc:374] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:34.065464 23390 raft_consensus.cc:729] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f53ee0561045969e4af5f80d088796, State: Initialized, Role: FOLLOWER
I20260620 23:01:34.065521 23390 consensus_queue.cc:260] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.065619 23390 ts_tablet_manager.cc:1428] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 23:01:34.065896 23391 tablet_bootstrap.cc:492] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: No bootstrap required, opened a new log
I20260620 23:01:34.065968 23391 ts_tablet_manager.cc:1397] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 23:01:34.066248 22870 catalog_manager.cc:4841] TS 8748ab320aaa4548ba51cea73d5f5e69 (127.21.66.65:42317): tablet 08f2a314c4524df796b7c612bf6e180e (table test-table [id=d22c2640f1804ddc881c41fbd1153a73]) successfully deleted
I20260620 23:01:34.066334 23391 raft_consensus.cc:348] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.066419 23391 raft_consensus.cc:374] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:34.066449 23391 raft_consensus.cc:729] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845bd6451bb24cf493281fab5cd0109e, State: Initialized, Role: FOLLOWER
I20260620 23:01:34.066505 23391 consensus_queue.cc:260] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.066636 23391 ts_tablet_manager.cc:1428] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:34.453675 23401 raft_consensus.cc:482] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:34.453792 23401 raft_consensus.cc:504] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.453981 23401 leader_election.cc:288] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 845bd6451bb24cf493281fab5cd0109e (127.21.66.66:45909), 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257)
I20260620 23:01:34.454300 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f449c9036a4247f1877a660950170d32" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f53ee0561045969e4af5f80d088796" is_pre_election: true
I20260620 23:01:34.454365 23181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f449c9036a4247f1877a660950170d32" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845bd6451bb24cf493281fab5cd0109e" is_pre_election: true
I20260620 23:01:34.454427 23321 raft_consensus.cc:2454] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 0.
I20260620 23:01:34.454466 23181 raft_consensus.cc:2454] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 0.
I20260620 23:01:34.454597 22973 leader_election.cc:302] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f53ee0561045969e4af5f80d088796, 8748ab320aaa4548ba51cea73d5f5e69; no voters:
I20260620 23:01:34.454721 23401 raft_consensus.cc:2781] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:34.454772 23401 raft_consensus.cc:482] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:34.454803 23401 raft_consensus.cc:3037] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:34.455626 23401 raft_consensus.cc:504] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.455829 23401 leader_election.cc:288] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 election: Requested vote from peers 845bd6451bb24cf493281fab5cd0109e (127.21.66.66:45909), 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257)
I20260620 23:01:34.455998 23181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f449c9036a4247f1877a660950170d32" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845bd6451bb24cf493281fab5cd0109e"
I20260620 23:01:34.455999 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f449c9036a4247f1877a660950170d32" candidate_uuid: "8748ab320aaa4548ba51cea73d5f5e69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f53ee0561045969e4af5f80d088796"
I20260620 23:01:34.456095 23181 raft_consensus.cc:3037] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:34.456095 23321 raft_consensus.cc:3037] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:34.456943 23181 raft_consensus.cc:2454] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 1.
I20260620 23:01:34.456993 23321 raft_consensus.cc:2454] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8748ab320aaa4548ba51cea73d5f5e69 in term 1.
I20260620 23:01:34.457158 22973 leader_election.cc:302] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f53ee0561045969e4af5f80d088796, 8748ab320aaa4548ba51cea73d5f5e69; no voters:
I20260620 23:01:34.457257 23401 raft_consensus.cc:2781] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:34.457299 23401 raft_consensus.cc:686] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 LEADER]: Becoming Leader. State: Replica: 8748ab320aaa4548ba51cea73d5f5e69, State: Running, Role: LEADER
I20260620 23:01:34.457340 23401 consensus_queue.cc:237] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.458143 22884 catalog_manager.cc:5495] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8748ab320aaa4548ba51cea73d5f5e69 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "8748ab320aaa4548ba51cea73d5f5e69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:01:34 dist-test-slave-qgtk krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996492, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:34.527076 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.518941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:41509 (local address 127.0.0.1:48244)
0620 23:01:34.519211 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:34.519223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:34.519299 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:34.519487 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:34.519488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:34.519874 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:34.519928 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.519930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.520828 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.520829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.521385 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.521387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.521758 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.521759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.521810 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.522432 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:34.522437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:34.524003 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:34.525746 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:34.525754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:34.525756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:34.525987 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:34.526341 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:34.526342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:34.526343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:34.526414 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:34.526738 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:34.526746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:34.526812 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 23:01:34.526879 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:01:34.526990 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260620 23:01:34.527221 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.519043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.21.66.126:41509)
0620 23:01:34.519216 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.519218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.519292 (+ 74us) server_negotiation.cc:407] Connection header received
0620 23:01:34.519360 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.519361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.519370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.519456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:34.520129 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.520740 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.521470 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.521667 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.521734 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.524088 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:34.524090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:34.524090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:34.524096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:34.525660 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:34.526197 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:34.526200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:34.526200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:34.526265 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:34.526505 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:34.526506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:34.526506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:34.526648 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:34.526712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.526910 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.527048 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260620 23:01:34.529551 22884 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:48244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:01:34.529749 22884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:34.531368 23161 tablet_service.cc:1467] Processing CreateTablet for tablet e85432851bf54409acf2e78ff91822bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f933f536a94f49e680569b0415319edb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:01:34.531375 23301 tablet_service.cc:1467] Processing CreateTablet for tablet e85432851bf54409acf2e78ff91822bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f933f536a94f49e680569b0415319edb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:01:34.531364 23021 tablet_service.cc:1467] Processing CreateTablet for tablet e85432851bf54409acf2e78ff91822bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f933f536a94f49e680569b0415319edb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:01:34.531550 23301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85432851bf54409acf2e78ff91822bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:34.531553 23021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85432851bf54409acf2e78ff91822bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:34.531553 23161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85432851bf54409acf2e78ff91822bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:34.532886 23391 tablet_bootstrap.cc:492] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: Bootstrap starting.
I20260620 23:01:34.533012 23394 tablet_bootstrap.cc:492] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: Bootstrap starting.
I20260620 23:01:34.533044 23390 tablet_bootstrap.cc:492] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: Bootstrap starting.
I20260620 23:01:34.533349 23391 tablet_bootstrap.cc:654] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:34.533366 23394 tablet_bootstrap.cc:654] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:34.533398 23390 tablet_bootstrap.cc:654] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:34.534047 23391 tablet_bootstrap.cc:492] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: No bootstrap required, opened a new log
I20260620 23:01:34.534109 23391 ts_tablet_manager.cc:1397] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:34.534178 23390 tablet_bootstrap.cc:492] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: No bootstrap required, opened a new log
I20260620 23:01:34.534230 23390 ts_tablet_manager.cc:1397] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:34.534258 23394 tablet_bootstrap.cc:492] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: No bootstrap required, opened a new log
I20260620 23:01:34.534308 23394 ts_tablet_manager.cc:1397] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:34.534313 23391 raft_consensus.cc:348] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.534399 23391 raft_consensus.cc:374] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:34.534379 23390 raft_consensus.cc:348] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.534428 23391 raft_consensus.cc:729] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845bd6451bb24cf493281fab5cd0109e, State: Initialized, Role: FOLLOWER
I20260620 23:01:34.534447 23390 raft_consensus.cc:374] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:34.534422 23394 raft_consensus.cc:348] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.534468 23390 raft_consensus.cc:729] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f53ee0561045969e4af5f80d088796, State: Initialized, Role: FOLLOWER
I20260620 23:01:34.534487 23394 raft_consensus.cc:374] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:34.534475 23391 consensus_queue.cc:260] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.534515 23394 raft_consensus.cc:729] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8748ab320aaa4548ba51cea73d5f5e69, State: Initialized, Role: FOLLOWER
I20260620 23:01:34.534508 23390 consensus_queue.cc:260] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.534569 23391 ts_tablet_manager.cc:1428] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:34.534556 23394 consensus_queue.cc:260] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.534622 23390 ts_tablet_manager.cc:1428] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:34.534650 23394 ts_tablet_manager.cc:1428] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:34.534703 23397 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.534742 23398 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.534789 23397 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:34.534802 23398 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:34.534879 23395 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.534955 23395 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:34.685285 23397 raft_consensus.cc:482] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:34.685400 23397 raft_consensus.cc:504] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.685871 23397 leader_election.cc:288] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8748ab320aaa4548ba51cea73d5f5e69 (127.21.66.65:42317), 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257)
I20260620 23:01:34.690255 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.686232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:40009 (local address 127.21.66.67:35257)
0620 23:01:34.686346 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.686348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.686358 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:34.686445 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.686446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.686458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.686520 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:34.686787 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.687548 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.689250 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.689894 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.689960 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.689968 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.690111 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.690178 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260620 23:01:34.690295 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.686158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:40719 (local address 127.21.66.65:42317)
0620 23:01:34.686315 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.686317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.686328 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:34.686412 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.686413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.686426 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.686510 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:34.686741 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.687583 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.689250 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.689845 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.689929 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.689942 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.690111 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.690221 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":58,"threads_started":1}
I20260620 23:01:34.690287 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.686092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:35257 (local address 127.21.66.66:40009)
0620 23:01:34.686276 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:34.686288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:34.686381 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:34.686563 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:34.686564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:34.686567 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:34.686672 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.686673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.687647 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.687648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.689155 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.689158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.689962 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.689963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.690031 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.690034 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:34.690093 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:01:34.690182 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":107,"thread_start_us":153,"threads_started":1}
I20260620 23:01:34.690565 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.686066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42317 (local address 127.21.66.66:40719)
0620 23:01:34.686258 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:34.686271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:34.686335 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:34.686563 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:34.686564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:34.686567 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:34.686656 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.686658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.687655 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.687656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.689155 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.689158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.689952 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.689953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.690025 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.690030 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:34.690093 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:01:34.690257 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":163,"thread_start_us":100,"threads_started":1}
I20260620 23:01:34.690548 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85432851bf54409acf2e78ff91822bd" candidate_uuid: "845bd6451bb24cf493281fab5cd0109e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f53ee0561045969e4af5f80d088796" is_pre_election: true
I20260620 23:01:34.690644 23321 raft_consensus.cc:2454] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 845bd6451bb24cf493281fab5cd0109e in term 0.
I20260620 23:01:34.690873 23113 leader_election.cc:302] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f53ee0561045969e4af5f80d088796, 845bd6451bb24cf493281fab5cd0109e; no voters:
I20260620 23:01:34.691015 23397 raft_consensus.cc:2781] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:34.690937 23041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85432851bf54409acf2e78ff91822bd" candidate_uuid: "845bd6451bb24cf493281fab5cd0109e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8748ab320aaa4548ba51cea73d5f5e69" is_pre_election: true
I20260620 23:01:34.691051 23397 raft_consensus.cc:482] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:34.691077 23397 raft_consensus.cc:3037] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:34.691079 23041 raft_consensus.cc:2454] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 845bd6451bb24cf493281fab5cd0109e in term 0.
I20260620 23:01:34.691829 23397 raft_consensus.cc:504] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.691989 23397 leader_election.cc:288] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [CANDIDATE]: Term 1 election: Requested vote from peers 8748ab320aaa4548ba51cea73d5f5e69 (127.21.66.65:42317), 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257)
I20260620 23:01:34.692133 23321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85432851bf54409acf2e78ff91822bd" candidate_uuid: "845bd6451bb24cf493281fab5cd0109e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57f53ee0561045969e4af5f80d088796"
I20260620 23:01:34.692214 23321 raft_consensus.cc:3037] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:34.692158 23041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85432851bf54409acf2e78ff91822bd" candidate_uuid: "845bd6451bb24cf493281fab5cd0109e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8748ab320aaa4548ba51cea73d5f5e69"
I20260620 23:01:34.692255 23041 raft_consensus.cc:3037] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:34.692813 23321 raft_consensus.cc:2454] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 845bd6451bb24cf493281fab5cd0109e in term 1.
I20260620 23:01:34.692968 23113 leader_election.cc:302] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57f53ee0561045969e4af5f80d088796, 845bd6451bb24cf493281fab5cd0109e; no voters:
I20260620 23:01:34.693050 23397 raft_consensus.cc:2781] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:34.693156 23397 raft_consensus.cc:686] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [term 1 LEADER]: Becoming Leader. State: Replica: 845bd6451bb24cf493281fab5cd0109e, State: Running, Role: LEADER
I20260620 23:01:34.693197 23041 raft_consensus.cc:2454] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 845bd6451bb24cf493281fab5cd0109e in term 1.
I20260620 23:01:34.693261 23397 consensus_queue.cc:237] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } }
I20260620 23:01:34.693739 23434 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: TxnStatusTablet state changed. Reason: New leader 845bd6451bb24cf493281fab5cd0109e. Latest consensus state: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.693861 23434 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:01:34.694057 23436 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:01:34.694022 22882 catalog_manager.cc:5495] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e reported cstate change: term changed from 0 to 1, leader changed from <none> to 845bd6451bb24cf493281fab5cd0109e (127.21.66.66). New cstate: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:34.694213 23436 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:01:34.721437 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.718682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.21.66.66:45909)
0620 23:01:34.718833 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.718835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.718843 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:34.718885 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.718886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.718922 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.719018 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:34.719256 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.719756 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.720512 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.720766 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.720844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.721038 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:34.721168 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:34.721251 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.721302 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.721324 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260620 23:01:34.721431 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.718577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:45909 (local address 127.0.0.1:59352)
0620 23:01:34.718694 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:34.718708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:34.718769 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:34.719013 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:34.719014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:34.719015 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:34.719084 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.719087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.719845 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.719846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.720426 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.720427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.720844 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.720845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.720915 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.720929 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:34.721237 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:34.721248 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:01:34.721299 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 23:01:34.721353 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:01:34.723027 23041 raft_consensus.cc:1261] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Refusing update from remote peer 845bd6451bb24cf493281fab5cd0109e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:34.723007 23321 raft_consensus.cc:1261] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Refusing update from remote peer 845bd6451bb24cf493281fab5cd0109e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:34.723302 23434 consensus_queue.cc:1035] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:34.723455 23397 consensus_queue.cc:1035] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:34.724002 23395 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: TxnStatusTablet state changed. Reason: New leader 845bd6451bb24cf493281fab5cd0109e. Latest consensus state: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.724097 23395 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:34.724157 23398 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: TxnStatusTablet state changed. Reason: New leader 845bd6451bb24cf493281fab5cd0109e. Latest consensus state: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.724260 23398 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:34.725029 23395 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.725034 23434 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.725116 23395 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 57f53ee0561045969e4af5f80d088796: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:34.725119 23434 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:01:34.726284 23434 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.726373 23434 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 845bd6451bb24cf493281fab5cd0109e: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:01:34.726442 23398 tablet_replica.cc:439] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "845bd6451bb24cf493281fab5cd0109e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8748ab320aaa4548ba51cea73d5f5e69" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42317 } } peers { permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 } } peers { permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 } } }
I20260620 23:01:34.726503 23443 mvcc.cc:205] Tried to move back new op lower bound from 7299057642382479360 to 7299057642263724032. Current Snapshot: MvccSnapshot[applied={T|T < 7299057642382479360}]
I20260620 23:01:34.726537 23398 tablet_replica.cc:442] T e85432851bf54409acf2e78ff91822bd P 8748ab320aaa4548ba51cea73d5f5e69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:34.731985 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.729348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:57465 (local address 127.21.66.66:45909)
0620 23:01:34.729403 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.729405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.729440 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:34.729518 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.729519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.729530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.729585 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:34.729742 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.730305 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.731103 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.731446 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.731538 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.731727 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:34.731761 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:34.731830 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.731892 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.731913 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:01:34.732023 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.729203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:45909 (local address 127.21.66.65:57465)
0620 23:01:34.729380 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:34.729394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:34.729458 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:34.729587 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:34.729587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:34.729589 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:34.729664 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.729666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.730393 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.730395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.731016 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.731018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.731542 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.731543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.731609 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.731629 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:34.731832 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:34.731834 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:01:34.731888 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:01:34.731939 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260620 23:01:34.734409 23321 raft_consensus.cc:1261] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Refusing update from remote peer 8748ab320aaa4548ba51cea73d5f5e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:34.734426 23181 raft_consensus.cc:1261] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Refusing update from remote peer 8748ab320aaa4548ba51cea73d5f5e69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:34.734614 23398 consensus_queue.cc:1035] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57f53ee0561045969e4af5f80d088796" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:34.734712 23401 consensus_queue.cc:1035] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "845bd6451bb24cf493281fab5cd0109e" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:34.746078 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.742694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:56807 (local address 127.21.66.65:42317)
0620 23:01:34.742744 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.742747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.742910 (+ 163us) server_negotiation.cc:407] Connection header received
0620 23:01:34.742971 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.742972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.742983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.743026 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:34.743249 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.743890 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.745036 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.745230 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.745294 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.745745 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:34.745789 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:34.745860 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.745949 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.745975 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:01:34.746368 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.742642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42317 (local address 127.21.66.66:56807)
0620 23:01:34.742835 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:34.742846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:34.742909 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:34.743030 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:34.743031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:34.743032 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:34.743111 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.743113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.743963 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.743964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.744922 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:34.744924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.745332 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.745334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:34.745399 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.745440 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:34.745862 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:34.745865 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:34.746218 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 23:01:34.746273 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":118,"threads_started":1}
I20260620 23:01:34.753643 22882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:34.753788 22882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:34.754599 22882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf6d93bc0df548479ae0f8b3ae5225e4: Sending DeleteTablet for 3 replicas of tablet f449c9036a4247f1877a660950170d32
I20260620 23:01:34.754994 23021 tablet_service.cc:1514] Processing DeleteTablet for tablet f449c9036a4247f1877a660950170d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:34 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:36850
I20260620 23:01:34.755214 23301 tablet_service.cc:1514] Processing DeleteTablet for tablet f449c9036a4247f1877a660950170d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:34 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52028
I20260620 23:01:34.755460 23161 tablet_service.cc:1514] Processing DeleteTablet for tablet f449c9036a4247f1877a660950170d32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:34 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:59344
I20260620 23:01:34.755488 23458 tablet_replica.cc:330] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: stopping tablet replica
I20260620 23:01:34.755548 23458 raft_consensus.cc:2229] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:34.755605 23458 raft_consensus.cc:2258] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:34.755673 23461 tablet_replica.cc:330] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: stopping tablet replica
I20260620 23:01:34.755728 23461 raft_consensus.cc:2229] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:34.755771 23461 raft_consensus.cc:2258] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:34.755976 23458 ts_tablet_manager.cc:1905] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:34.756011 23461 ts_tablet_manager.cc:1905] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:34.755986 23460 tablet_replica.cc:330] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: stopping tablet replica
I20260620 23:01:34.756140 23460 raft_consensus.cc:2229] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:34.756228 23460 raft_consensus.cc:2258] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:34.756568 23460 ts_tablet_manager.cc:1905] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:34.757879 23461 ts_tablet_manager.cc:1918] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:34.757949 23461 log.cc:1196] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/f449c9036a4247f1877a660950170d32
I20260620 23:01:34.758258 23460 ts_tablet_manager.cc:1918] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:34.758316 23460 log.cc:1196] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/f449c9036a4247f1877a660950170d32
I20260620 23:01:34.758549 23460 ts_tablet_manager.cc:1939] T f449c9036a4247f1877a660950170d32 P 57f53ee0561045969e4af5f80d088796: Deleting consensus metadata
I20260620 23:01:34.758561 23461 ts_tablet_manager.cc:1939] T f449c9036a4247f1877a660950170d32 P 845bd6451bb24cf493281fab5cd0109e: Deleting consensus metadata
I20260620 23:01:34.758875 22869 catalog_manager.cc:4841] TS 57f53ee0561045969e4af5f80d088796 (127.21.66.67:35257): tablet f449c9036a4247f1877a660950170d32 (table test-table [id=359602d393834dfd9cb5fd04ba142fae]) successfully deleted
I20260620 23:01:34.758942 22870 catalog_manager.cc:4841] TS 845bd6451bb24cf493281fab5cd0109e (127.21.66.66:45909): tablet f449c9036a4247f1877a660950170d32 (table test-table [id=359602d393834dfd9cb5fd04ba142fae]) successfully deleted
I20260620 23:01:34.760459 23458 ts_tablet_manager.cc:1918] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:34.760551 23458 log.cc:1196] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/f449c9036a4247f1877a660950170d32
I20260620 23:01:34.760816 23458 ts_tablet_manager.cc:1939] T f449c9036a4247f1877a660950170d32 P 8748ab320aaa4548ba51cea73d5f5e69: Deleting consensus metadata
I20260620 23:01:34.761138 22870 catalog_manager.cc:4841] TS 8748ab320aaa4548ba51cea73d5f5e69 (127.21.66.65:42317): tablet f449c9036a4247f1877a660950170d32 (table test-table [id=359602d393834dfd9cb5fd04ba142fae]) successfully deleted
Jun 20 23:01:34 dist-test-slave-qgtk krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:34.766503 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.755625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.21.66.65:42317)
0620 23:01:34.755728 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.755730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.756442 (+ 712us) server_negotiation.cc:407] Connection header received
0620 23:01:34.756556 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.756557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.756571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.756601 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:34.757413 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.758176 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.759424 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.759685 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.759783 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.762911 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:34.762921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:34.762921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:34.762928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:34.764973 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:34.765452 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:34.765456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:34.765456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:34.765529 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:34.765757 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:34.765758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:34.765758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:34.765916 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:34.766032 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.766196 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.766386 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:01:34.766778 23081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36876
I20260620 23:01:34.774695 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:34.767858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.21.66.126:41509)
0620 23:01:34.767932 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:34.767935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:34.768070 (+ 135us) server_negotiation.cc:407] Connection header received
0620 23:01:34.768150 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:34.768151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:34.768161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:34.768207 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:34.768732 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.769238 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.769967 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:34.770274 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:34.770361 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:34.771638 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:34.771640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:34.771640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:34.771648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:34.773251 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:34.773668 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:34.773669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:34.773670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:34.773737 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:34.773946 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:34.773947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:34.773948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:34.774082 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:34.774269 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:34.774471 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 23:01:34.774569 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 23:01:34.774993 22882 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48258
I20260620 23:01:34.775518 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22956
I20260620 23:01:34.784349 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 23096
I20260620 23:01:34.789726 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 23236
I20260620 23:01:34.794711 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 22852
2026-06-20T23:01:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3130 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:34 dist-test-slave-qgtk krb5kdc[23467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:34 dist-test-slave-qgtk krb5kdc[23467](info): set up 2 sockets
Jun 20 23:01:34 dist-test-slave-qgtk krb5kdc[23467](info): commencing operation
krb5kdc: starting...
W20260620 23:01:35.902840 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:35 dist-test-slave-qgtk krb5kdc[23467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:36.038774 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39291
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:36.130386 23480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:36.130682 23480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:36.130729 23480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:36.132159 23480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:36.132249 23480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:01:36.132274 23480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:36.132297 23480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:36.132331 23480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:36.132364 23480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:36.134348 23480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39291
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:36.134718 23480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:36.135015 23480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:36.140791 23485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:36.140827 23486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:36.141008 23480 server_base.cc:1034] running on GCE node
W20260620 23:01:36.140791 23488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:36.141348 23480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:36.141716 23480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:36.142905 23480 hybrid_clock.cc:648] HybridClock initialized: now 1781996496142867 us; error 68 us; skew 500 ppm
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:36.145864 23480 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:36.146431 23480 webserver.cc:460] Webserver started at http://127.21.66.126:38899/ using document root <none> and password file <none>
I20260620 23:01:36.146746 23480 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:36.146818 23480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:36.146976 23480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:36.148109 23480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "1e160ef6515947699af602da657fa80d"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "2acd29120faec6f743146b8f5371408c"
server_key_iv: "14ccc49db3dbe5579df391850ba9e4d0"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.148507 23480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "1e160ef6515947699af602da657fa80d"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "2acd29120faec6f743146b8f5371408c"
server_key_iv: "14ccc49db3dbe5579df391850ba9e4d0"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.150182 23480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:36.151252 23495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.151500 23480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:36.151635 23480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "1e160ef6515947699af602da657fa80d"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "2acd29120faec6f743146b8f5371408c"
server_key_iv: "14ccc49db3dbe5579df391850ba9e4d0"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.151731 23480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:36.170491 23480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:36.172992 23480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:36.173213 23480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:36.178500 23480 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:39291
I20260620 23:01:36.178614 23557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:39291 every 8 connection(s)
I20260620 23:01:36.178961 23480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:36.179615 23558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:36.181977 23558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Bootstrap starting.
I20260620 23:01:36.182622 23558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:36.182929 23558 log.cc:824] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:36.183769 23558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: No bootstrap required, opened a new log
I20260620 23:01:36.185060 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 23480
I20260620 23:01:36.185199 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:36.185392 21769 external_mini_cluster.cc:1428] Setting key 00e703382584ecdd693e41a5795b6aa6
I20260620 23:01:36.186466 23558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e160ef6515947699af602da657fa80d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39291 } }
I20260620 23:01:36.186653 23558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:36.186708 23558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e160ef6515947699af602da657fa80d, State: Initialized, Role: FOLLOWER
I20260620 23:01:36.186885 23558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e160ef6515947699af602da657fa80d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39291 } }
I20260620 23:01:36.186975 23558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:36.187000 23558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:36.187054 23558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:36.187775 23558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e160ef6515947699af602da657fa80d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39291 } }
I20260620 23:01:36.187891 23558 leader_election.cc:302] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e160ef6515947699af602da657fa80d; no voters:
I20260620 23:01:36.188055 23558 leader_election.cc:288] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:36.188159 23563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:36.188344 23558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:36.188357 23563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [term 1 LEADER]: Becoming Leader. State: Replica: 1e160ef6515947699af602da657fa80d, State: Running, Role: LEADER
I20260620 23:01:36.188613 23563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e160ef6515947699af602da657fa80d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39291 } }
I20260620 23:01:36.189029 23565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e160ef6515947699af602da657fa80d. Latest consensus state: current_term: 1 leader_uuid: "1e160ef6515947699af602da657fa80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e160ef6515947699af602da657fa80d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39291 } } }
I20260620 23:01:36.189109 23565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:36.189561 23564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e160ef6515947699af602da657fa80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e160ef6515947699af602da657fa80d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39291 } } }
I20260620 23:01:36.189640 23564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:36.189951 23572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:36.190505 23572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:36.192133 23572 catalog_manager.cc:1347] Generated new cluster ID: 57c7d2fc28c24dcfbf42832af74d5ad5
I20260620 23:01:36.192190 23572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:36.198422 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.186038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.21.66.126:39291)
0620 23:01:36.186306 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.186318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.186357 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:01:36.186482 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.186490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.186549 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.186719 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.187610 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.189076 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.190330 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.190920 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.191042 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.194155 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.194159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.194160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.194173 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.196442 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.196950 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.196955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.196955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.197097 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.197336 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.197336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.197337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.197663 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.197828 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.197960 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.198151 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":146,"threads_started":1}
I20260620 23:01:36.198498 23572 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:36.199471 23572 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:36.203410 23572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Generated new TSK 0
I20260620 23:01:36.203637 23572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:36.234067 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39291
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:36.328653 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:36.328961 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:36.329013 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:36.330461 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:36.330559 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:36.330598 23584 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:01:36.330636 23584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:01:36.330694 23584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:36.332496 23584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:39291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:36.332831 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:36.333122 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:36.339454 23590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:36.339468 23589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:36.339453 23592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:36.339573 23584 server_base.cc:1034] running on GCE node
I20260620 23:01:36.340004 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:36.340317 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:36.341527 23584 hybrid_clock.cc:648] HybridClock initialized: now 1781996496341507 us; error 47 us; skew 500 ppm
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:36.344547 23584 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:36.345144 23584 webserver.cc:460] Webserver started at http://127.21.66.65:38723/ using document root <none> and password file <none>
I20260620 23:01:36.345446 23584 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:36.345527 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:36.345700 23584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:36.347110 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "51ce103b7688479c822045a26e935121"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "59657946a6d98e7b90a26027a2dfcdee"
server_key_iv: "c128ab8d82687786601dd84a6df117a5"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.347543 23584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "51ce103b7688479c822045a26e935121"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "59657946a6d98e7b90a26027a2dfcdee"
server_key_iv: "c128ab8d82687786601dd84a6df117a5"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.349157 23584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:36.350039 23599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.350231 23584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:36.350344 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "51ce103b7688479c822045a26e935121"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "59657946a6d98e7b90a26027a2dfcdee"
server_key_iv: "c128ab8d82687786601dd84a6df117a5"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.350467 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:36.378602 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:36.381256 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:36.381510 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:36.382301 23584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:36.382364 23584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.382411 23584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:36.382447 23584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:36.390806 23584 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:34383
I20260620 23:01:36.390923 23714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:34383 every 8 connection(s)
I20260620 23:01:36.391330 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:36.394444 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.382789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:41793 (local address 127.21.66.126:39291)
0620 23:01:36.382871 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.382873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.383149 (+ 276us) server_negotiation.cc:407] Connection header received
0620 23:01:36.383679 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.383681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.383700 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.383789 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.384689 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.385676 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.387025 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.387429 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.387532 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.391230 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.391231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.391232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.391242 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.392857 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.393385 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.393386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.393387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.393465 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.393720 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.393721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.393721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.393844 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.393904 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.394274 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.394364 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:01:36.394601 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.382787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39291 (local address 127.21.66.65:41793)
0620 23:01:36.382989 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.383035 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.383573 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.383889 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.383896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.384294 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:36.384592 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.384599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.385812 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.385813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.386898 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.386908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.387555 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.387556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.387711 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.388375 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:36.388387 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:36.391092 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:36.392952 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.392958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.392960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.393278 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.393549 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.393550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.393550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.393657 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.393892 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:36.393895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:36.394076 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.394230 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.394376 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":124,"threads_started":1}
I20260620 23:01:36.395196 23715 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39291
I20260620 23:01:36.395296 23715 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:36.395494 23715 heartbeater.cc:511] Master 127.21.66.126:39291 requested a full tablet report, sending...
I20260620 23:01:36.396098 23512 ts_manager.cc:194] Registered new tserver with Master: 51ce103b7688479c822045a26e935121 (127.21.66.65:34383)
I20260620 23:01:36.396840 23512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:41793
I20260620 23:01:36.400563 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 23584
I20260620 23:01:36.400704 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:36.400923 21769 external_mini_cluster.cc:1428] Setting key 734f536c8cf3a451ba884a0d88f5e7c4
I20260620 23:01:36.403663 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.396508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:51337 (local address 127.21.66.126:39291)
0620 23:01:36.397051 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.397053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.397064 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:36.397087 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.397087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.397095 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.397143 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.397564 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.398203 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.398993 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.399195 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.399286 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.400522 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.400524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.400524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.400534 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.402279 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.402740 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.402741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.402742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.402803 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.403067 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.403068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.403068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.403182 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.403280 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.403460 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.403565 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515}
I20260620 23:01:36.403678 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.396442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39291 (local address 127.21.66.65:51337)
0620 23:01:36.396571 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.396582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.396636 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.397154 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.397155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.397412 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:36.397476 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.397477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.398297 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.398298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.398884 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.398886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.399287 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.399288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.399379 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.399899 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:36.399911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:36.400246 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:36.402360 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.402361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.402362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.402630 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.402880 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.402880 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.402881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.402955 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.403287 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:36.403288 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:36.403343 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.403433 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.403562 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:36.431139 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39291
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:36.524497 23724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:36.524746 23724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:36.524777 23724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:36.526207 23724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:36.526262 23724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:36.526288 23724 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:01:36.526340 23724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:01:36.526432 23724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:36.528292 23724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:39291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:36.528631 23724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:36.528916 23724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:36.534960 23729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:36.534983 23730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:36.535168 23732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:36.535413 23724 server_base.cc:1034] running on GCE node
I20260620 23:01:36.535688 23724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:36.536077 23724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:36.537281 23724 hybrid_clock.cc:648] HybridClock initialized: now 1781996496537267 us; error 31 us; skew 500 ppm
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:36.540259 23724 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:36.540783 23724 webserver.cc:460] Webserver started at http://127.21.66.66:38283/ using document root <none> and password file <none>
I20260620 23:01:36.541062 23724 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:36.541134 23724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:36.541287 23724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:36.542484 23724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "5ff577bb049445fe957434f4da166586"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "622a961864324dfbd568a99f0326b5ee"
server_key_iv: "e5f6538e7691aed02abdb3c97fefbc64"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.542873 23724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "5ff577bb049445fe957434f4da166586"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "622a961864324dfbd568a99f0326b5ee"
server_key_iv: "e5f6538e7691aed02abdb3c97fefbc64"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.544270 23724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:36.545161 23739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.545490 23724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:36.545567 23724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "5ff577bb049445fe957434f4da166586"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "622a961864324dfbd568a99f0326b5ee"
server_key_iv: "e5f6538e7691aed02abdb3c97fefbc64"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.545624 23724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:36.565506 23724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:36.568328 23724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:36.568502 23724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:36.569023 23724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:36.569064 23724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.569089 23724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:36.569103 23724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.576398 23724 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:41717
I20260620 23:01:36.576444 23854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:41717 every 8 connection(s)
I20260620 23:01:36.576850 23724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:36.577349 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 23724
I20260620 23:01:36.577451 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:36.577648 21769 external_mini_cluster.cc:1428] Setting key 4800bc324e1867d1ff4283b5290c9fc4
I20260620 23:01:36.581251 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.569676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:56625 (local address 127.21.66.126:39291)
0620 23:01:36.569721 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.569723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.569942 (+ 219us) server_negotiation.cc:407] Connection header received
0620 23:01:36.570593 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.570594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.570606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.570654 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.571547 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.572386 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.574220 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.574512 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.574579 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.577731 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.577733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.577734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.577784 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.579666 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.580149 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.580150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.580151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.580233 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.580511 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.580512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.580513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.580681 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.580775 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.581067 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.581178 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:36.581540 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.569609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39291 (local address 127.21.66.66:56625)
0620 23:01:36.569817 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.569855 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.570472 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.570730 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.570734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.571137 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:36.571448 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.571454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.572493 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.572495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.574048 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.574051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.574612 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.574613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.574791 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.575529 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:36.575540 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:36.577606 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:36.579768 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.579779 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.579782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.580064 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.580314 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.580314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.580315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.580442 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.580774 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:36.580779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:36.580904 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.581030 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.581215 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260620 23:01:36.582435 23855 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39291
I20260620 23:01:36.582549 23855 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:36.582800 23855 heartbeater.cc:511] Master 127.21.66.126:39291 requested a full tablet report, sending...
I20260620 23:01:36.583470 23512 ts_manager.cc:194] Registered new tserver with Master: 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717)
I20260620 23:01:36.584023 23512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:56625
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
I20260620 23:01:36.590032 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.583605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:52543 (local address 127.21.66.126:39291)
0620 23:01:36.583654 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.583656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.583903 (+ 247us) server_negotiation.cc:407] Connection header received
0620 23:01:36.583934 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.583936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.583947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.584020 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.584448 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.584929 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.585641 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.585849 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.585936 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.587056 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.587057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.587058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.587066 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.588478 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.588957 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.588958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.588959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.589029 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.589250 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.589251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.589252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.589499 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.589599 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.589804 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.589904 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:01:36.590227 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.583657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39291 (local address 127.21.66.66:52543)
0620 23:01:36.583819 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.583830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.583891 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.584026 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.584026 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.584260 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:36.584340 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.584343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.585037 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.585038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.585554 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.585555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.585943 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.585944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.586046 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.586591 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:36.586596 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:36.586968 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:36.588603 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.588604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.588604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.588858 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.589110 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.589111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.589111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.589166 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.589614 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:36.589615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:36.589694 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.589779 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.589905 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:36.608788 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39291
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:36.703413 23864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:36.703698 23864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:36.703747 23864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:36.705081 23864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:36.705161 23864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:36.705204 23864 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:01:36.705255 23864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:01:36.705299 23864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:36.707103 23864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:39291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:36.707427 23864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:36.707697 23864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:36.713171 23872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:36.713222 23869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:36.713306 23870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:36.713323 23864 server_base.cc:1034] running on GCE node
I20260620 23:01:36.713913 23864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:36.714260 23864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:36.715408 23864 hybrid_clock.cc:648] HybridClock initialized: now 1781996496715393 us; error 38 us; skew 500 ppm
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:36.717905 23864 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:36.718472 23864 webserver.cc:460] Webserver started at http://127.21.66.67:41555/ using document root <none> and password file <none>
I20260620 23:01:36.718724 23864 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:36.718797 23864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:36.718963 23864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:36.720086 23864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "7fd91de572134a018ee2c46763a3a889"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "1ebba1bc3bfd7419f3ee4503eb97ac9a"
server_key_iv: "eb7935d1fa5c8fdc2eb5cb8f82d09e57"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.720455 23864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "7fd91de572134a018ee2c46763a3a889"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "1ebba1bc3bfd7419f3ee4503eb97ac9a"
server_key_iv: "eb7935d1fa5c8fdc2eb5cb8f82d09e57"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.721894 23864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:36.722800 23879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.722977 23864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:36.723160 23864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "7fd91de572134a018ee2c46763a3a889"
format_stamp: "Formatted at 2026-06-20 23:01:36 on dist-test-slave-qgtk"
server_key: "1ebba1bc3bfd7419f3ee4503eb97ac9a"
server_key_iv: "eb7935d1fa5c8fdc2eb5cb8f82d09e57"
server_key_version: "encryptionkey@0"
I20260620 23:01:36.723249 23864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:36.732117 23864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:36.734860 23864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:36.735042 23864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:36.735616 23864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:36.735685 23864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.735735 23864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:36.735764 23864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.742640 23864 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:41797
I20260620 23:01:36.742853 23994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:41797 every 8 connection(s)
I20260620 23:01:36.743124 23864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:36.744443 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 23864
I20260620 23:01:36.744516 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:36.744690 21769 external_mini_cluster.cc:1428] Setting key 34918b9611d75e33d9c46f29c1bd86b0
I20260620 23:01:36.747605 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.736127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:38049 (local address 127.21.66.126:39291)
0620 23:01:36.736198 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.736201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.736510 (+ 309us) server_negotiation.cc:407] Connection header received
0620 23:01:36.737189 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.737190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.737206 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.737283 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.738283 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.739196 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.740933 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.741293 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.741382 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.744362 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.744364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.744365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.744376 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.746105 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.746560 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.746561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.746562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.746631 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.746906 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.746907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.746907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.747044 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.747120 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.747423 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.747512 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:01:36.747834 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.736184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39291 (local address 127.21.66.67:38049)
0620 23:01:36.736363 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.736407 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.737039 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.737381 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.737387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.737798 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:36.738107 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.738174 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.739320 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.739321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.740816 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.740820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.741390 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.741391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.741605 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.742350 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:36.742359 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:36.744233 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:36.746210 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.746218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.746220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.746462 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.746702 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.746703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.746704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.746827 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.747112 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:36.747119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:36.747275 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.747388 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.747576 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260620 23:01:36.748342 23995 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39291
I20260620 23:01:36.748430 23995 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:36.748656 23995 heartbeater.cc:511] Master 127.21.66.126:39291 requested a full tablet report, sending...
I20260620 23:01:36.749429 23512 ts_manager.cc:194] Registered new tserver with Master: 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797)
I20260620 23:01:36.749994 23512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:38049
I20260620 23:01:36.756315 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.749486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:55211 (local address 127.21.66.126:39291)
0620 23:01:36.749528 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.749530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.749671 (+ 141us) server_negotiation.cc:407] Connection header received
0620 23:01:36.749739 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.749740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.749748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.749795 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.750241 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.750975 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.751723 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.751898 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.751945 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.753007 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.753009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.753010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.753018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.754884 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.755324 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.755325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.755326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.755381 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.755639 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.755640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.755641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.755818 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.755879 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.756098 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.756215 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:36.756340 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.749434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39291 (local address 127.21.66.67:55211)
0620 23:01:36.749608 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.749620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.749678 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.749800 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.749800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.749995 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:36.750073 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.750075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.751073 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.751074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.751652 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.751654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.751958 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.751958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.752010 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.752483 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:36.752488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:36.752825 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:36.754970 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.754971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.754972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.755234 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.755442 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.755442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.755443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.755541 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.755921 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:36.755922 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:36.755971 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.756071 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.756199 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260620 23:01:36.756817 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:36.781109 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.773158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.21.66.126:39291)
0620 23:01:36.773225 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.773227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.773244 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:01:36.773272 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.773272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.773281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.773333 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.773965 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.774608 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.775415 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.775606 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.775670 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.778383 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.778384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.778385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.778393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.780121 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.780475 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.780476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.780477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.780527 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.780690 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.780691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.780691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.780801 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.780872 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.780978 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.781037 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:01:36.783069 23512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:36.784986 23512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:36.792840 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.788513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.21.66.65:34383)
0620 23:01:36.788696 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.788701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.788976 (+ 275us) server_negotiation.cc:407] Connection header received
0620 23:01:36.789024 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.789026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.789044 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.789149 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:36.789275 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.790047 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.791870 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.792377 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.792476 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.792540 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.792683 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.792724 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260620 23:01:36.792887 24012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.788028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:41717 (local address 127.0.0.1:57384)
0620 23:01:36.788255 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.788313 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.788424 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.789051 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.789052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.789054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:36.789237 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.789242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.790618 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.790620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.791467 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.791470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.792527 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.792528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.792603 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.792608 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.792642 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.792757 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260620 23:01:36.793056 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.788125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.21.66.67:41797)
0620 23:01:36.788498 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.788504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.788518 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:01:36.788570 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.788574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.788596 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.788734 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:36.788943 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.790270 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.791732 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.792522 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.792588 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.792651 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.792928 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.792974 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":94,"threads_started":1}
I20260620 23:01:36.793229 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.788029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41797 (local address 127.0.0.1:42744)
0620 23:01:36.788419 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.788431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.788490 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.788747 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.788751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.788754 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:36.788846 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.788849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.790357 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.790358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.791636 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.791639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.792686 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.792687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.792771 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.792775 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.792829 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.792869 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":58,"thread_start_us":121,"threads_started":1}
I20260620 23:01:36.793246 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.788289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34383 (local address 127.0.0.1:50072)
0620 23:01:36.788863 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.788872 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.788921 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.789157 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.789158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.789160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:36.789202 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.789204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.790209 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.790210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.791784 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.791788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.792480 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.792481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.792564 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.792578 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.792668 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.792740 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":58,"thread_start_us":69,"threads_started":1}
I20260620 23:01:36.793270 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.788138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.21.66.66:41717)
0620 23:01:36.788880 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.788885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.788896 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:36.788939 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.788941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.788957 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.789047 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:36.789382 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.790531 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.791585 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.792411 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.793012 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.793083 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.793112 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.793162 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":90,"threads_started":1}
I20260620 23:01:36.793944 23789 tablet_service.cc:1467] Processing CreateTablet for tablet b147ce35eae040e58fc6fd06333dda99 (DEFAULT_TABLE table=test-table [id=a4478f2f16dd4554860a6430686235b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:36.793993 23929 tablet_service.cc:1467] Processing CreateTablet for tablet b147ce35eae040e58fc6fd06333dda99 (DEFAULT_TABLE table=test-table [id=a4478f2f16dd4554860a6430686235b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:36.794185 23649 tablet_service.cc:1467] Processing CreateTablet for tablet b147ce35eae040e58fc6fd06333dda99 (DEFAULT_TABLE table=test-table [id=a4478f2f16dd4554860a6430686235b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:36.794366 23789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b147ce35eae040e58fc6fd06333dda99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:36.794407 23929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b147ce35eae040e58fc6fd06333dda99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:36.794471 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b147ce35eae040e58fc6fd06333dda99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:36.797626 24018 tablet_bootstrap.cc:492] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Bootstrap starting.
I20260620 23:01:36.798223 24019 tablet_bootstrap.cc:492] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Bootstrap starting.
I20260620 23:01:36.798287 24020 tablet_bootstrap.cc:492] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Bootstrap starting.
I20260620 23:01:36.798435 24018 tablet_bootstrap.cc:654] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:36.798764 24018 log.cc:824] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:36.798939 24020 tablet_bootstrap.cc:654] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:36.799214 24019 tablet_bootstrap.cc:654] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:36.799285 24020 log.cc:824] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:36.799548 24019 log.cc:824] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:36.799798 24018 tablet_bootstrap.cc:492] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: No bootstrap required, opened a new log
I20260620 23:01:36.799901 24018 ts_tablet_manager.cc:1397] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:36.800102 24020 tablet_bootstrap.cc:492] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: No bootstrap required, opened a new log
I20260620 23:01:36.800192 24020 ts_tablet_manager.cc:1397] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:36.801719 24020 raft_consensus.cc:348] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.801908 24020 raft_consensus.cc:374] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:36.801957 24020 raft_consensus.cc:729] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ce103b7688479c822045a26e935121, State: Initialized, Role: FOLLOWER
I20260620 23:01:36.802084 24020 consensus_queue.cc:260] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.801962 24018 raft_consensus.cc:348] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.802235 24018 raft_consensus.cc:374] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:36.802289 24018 raft_consensus.cc:729] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff577bb049445fe957434f4da166586, State: Initialized, Role: FOLLOWER
I20260620 23:01:36.802428 24020 ts_tablet_manager.cc:1428] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:01:36.802438 24018 consensus_queue.cc:260] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.802562 23715 heartbeater.cc:503] Master 127.21.66.126:39291 was elected leader, sending a full tablet report...
I20260620 23:01:36.802743 24018 ts_tablet_manager.cc:1428] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:36.802794 23855 heartbeater.cc:503] Master 127.21.66.126:39291 was elected leader, sending a full tablet report...
I20260620 23:01:36.802855 24019 tablet_bootstrap.cc:492] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: No bootstrap required, opened a new log
I20260620 23:01:36.802965 24019 ts_tablet_manager.cc:1397] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 23:01:36.804939 24019 raft_consensus.cc:348] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.805073 24019 raft_consensus.cc:374] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:36.805095 24019 raft_consensus.cc:729] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd91de572134a018ee2c46763a3a889, State: Initialized, Role: FOLLOWER
I20260620 23:01:36.805190 24019 consensus_queue.cc:260] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.805390 24019 ts_tablet_manager.cc:1428] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:01:36.805519 23995 heartbeater.cc:503] Master 127.21.66.126:39291 was elected leader, sending a full tablet report...
W20260620 23:01:36.828163 23856 tablet.cc:2369] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:01:36.892467 23716 tablet.cc:2369] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:01:36.912675 24025 raft_consensus.cc:482] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:36.912810 24025 raft_consensus.cc:504] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.913298 24025 leader_election.cc:288] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797), 51ce103b7688479c822045a26e935121 (127.21.66.65:34383)
I20260620 23:01:36.916448 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.913780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:45621 (local address 127.21.66.67:41797)
0620 23:01:36.913831 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.913839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.913848 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:36.913873 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.913874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.913881 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.913963 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:36.914229 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.914702 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.915682 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.916053 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.916166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.916175 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.916349 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.916383 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:01:36.916446 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.913770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:40403 (local address 127.21.66.65:34383)
0620 23:01:36.913826 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.913828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.913927 (+ 99us) server_negotiation.cc:407] Connection header received
0620 23:01:36.914018 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.914019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.914028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.914093 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:36.914285 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.914750 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.915709 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.916067 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.916168 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.916177 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.916336 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.916374 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:01:36.916460 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.913538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41797 (local address 127.21.66.66:45621)
0620 23:01:36.913650 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.913664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.913754 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.913968 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.913969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.913971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:36.914045 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.914047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.914832 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.914834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.915579 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.915581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.916171 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.916172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.916263 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.916266 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.916340 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.916380 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:01:36.916468 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.913679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34383 (local address 127.21.66.66:40403)
0620 23:01:36.913856 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.913865 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.913936 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.914095 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.914096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.914098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:36.914212 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.914215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.914832 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.914833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.915596 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.915598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.916168 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.916169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.916248 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.916251 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.916327 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.916369 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260620 23:01:36.916899 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147ce35eae040e58fc6fd06333dda99" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd91de572134a018ee2c46763a3a889" is_pre_election: true
I20260620 23:01:36.916882 23669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147ce35eae040e58fc6fd06333dda99" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121" is_pre_election: true
I20260620 23:01:36.917052 23669 raft_consensus.cc:2454] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 0.
I20260620 23:01:36.917050 23949 raft_consensus.cc:2454] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 0.
I20260620 23:01:36.917344 23742 leader_election.cc:302] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce103b7688479c822045a26e935121, 5ff577bb049445fe957434f4da166586; no voters:
I20260620 23:01:36.917476 24025 raft_consensus.cc:2781] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:36.917536 24025 raft_consensus.cc:482] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:36.917572 24025 raft_consensus.cc:3037] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:36.918358 24025 raft_consensus.cc:504] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.918500 24025 leader_election.cc:288] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 election: Requested vote from peers 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797), 51ce103b7688479c822045a26e935121 (127.21.66.65:34383)
I20260620 23:01:36.918696 23669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147ce35eae040e58fc6fd06333dda99" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121"
I20260620 23:01:36.918690 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b147ce35eae040e58fc6fd06333dda99" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd91de572134a018ee2c46763a3a889"
I20260620 23:01:36.918779 23949 raft_consensus.cc:3037] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:36.918776 23669 raft_consensus.cc:3037] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:36.919492 23949 raft_consensus.cc:2454] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 1.
I20260620 23:01:36.919534 23669 raft_consensus.cc:2454] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 1.
I20260620 23:01:36.919695 23742 leader_election.cc:302] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff577bb049445fe957434f4da166586, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:36.919802 24025 raft_consensus.cc:2781] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:36.919943 24025 raft_consensus.cc:686] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff577bb049445fe957434f4da166586, State: Running, Role: LEADER
I20260620 23:01:36.920043 24025 consensus_queue.cc:237] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } }
I20260620 23:01:36.920936 23511 catalog_manager.cc:5495] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff577bb049445fe957434f4da166586 (127.21.66.66). New cstate: current_term: 1 leader_uuid: "5ff577bb049445fe957434f4da166586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:01:36 dist-test-slave-qgtk krb5kdc[23467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996496, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:36.966053 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.958914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39291 (local address 127.0.0.1:39856)
0620 23:01:36.959152 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:36.959161 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:36.959252 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:36.959407 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:36.959408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:36.959733 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:36.959783 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.959786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.960436 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.960437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.960969 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:36.960971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.961325 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.961326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:36.961376 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.961920 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:36.961924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:36.963501 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:36.965065 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.965073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.965075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.965266 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.965447 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:36.965448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:36.965448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:36.965502 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:36.965716 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:36.965722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:36.965763 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:01:36.965838 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:01:36.965936 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":151,"threads_started":1}
I20260620 23:01:36.966063 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:36.959003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.21.66.126:39291)
0620 23:01:36.959091 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:36.959093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:36.959232 (+ 139us) server_negotiation.cc:407] Connection header received
0620 23:01:36.959322 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:36.959323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:36.959334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:36.959403 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:36.959869 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.960352 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.961067 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:36.961247 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:36.961328 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:36.963597 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:36.963598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:36.963599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:36.963606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:36.964986 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.965334 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.965335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.965336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.965387 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:36.965553 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:36.965553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:36.965554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:36.965663 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:36.965742 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:36.965870 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:01:36.965980 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":36}
I20260620 23:01:36.968571 23511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:39856:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:01:36.968775 23511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:36.970568 23649 tablet_service.cc:1467] Processing CreateTablet for tablet bc7d28bf09e64c00995d6dd7303ec096 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6537aa64b4f84c5f95ac4d5b0b43a633]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:01:36.970669 23789 tablet_service.cc:1467] Processing CreateTablet for tablet bc7d28bf09e64c00995d6dd7303ec096 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6537aa64b4f84c5f95ac4d5b0b43a633]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:01:36.970770 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7d28bf09e64c00995d6dd7303ec096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:36.970814 23789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7d28bf09e64c00995d6dd7303ec096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:36.972461 24018 tablet_bootstrap.cc:492] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: Bootstrap starting.
I20260620 23:01:36.972515 23929 tablet_service.cc:1467] Processing CreateTablet for tablet bc7d28bf09e64c00995d6dd7303ec096 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6537aa64b4f84c5f95ac4d5b0b43a633]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:01:36.972720 23929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7d28bf09e64c00995d6dd7303ec096. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:36.973109 24018 tablet_bootstrap.cc:654] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:36.973515 24020 tablet_bootstrap.cc:492] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: Bootstrap starting.
I20260620 23:01:36.973876 24018 tablet_bootstrap.cc:492] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: No bootstrap required, opened a new log
I20260620 23:01:36.973948 24020 tablet_bootstrap.cc:654] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:36.973935 24018 ts_tablet_manager.cc:1397] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:36.974169 24019 tablet_bootstrap.cc:492] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: Bootstrap starting.
I20260620 23:01:36.974179 24018 raft_consensus.cc:348] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:36.974287 24018 raft_consensus.cc:374] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:36.974316 24018 raft_consensus.cc:729] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff577bb049445fe957434f4da166586, State: Initialized, Role: FOLLOWER
I20260620 23:01:36.974403 24018 consensus_queue.cc:260] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:36.974519 24018 ts_tablet_manager.cc:1428] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:36.974603 24025 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:36.974725 24025 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:36.974778 24019 tablet_bootstrap.cc:654] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:36.975471 24020 tablet_bootstrap.cc:492] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: No bootstrap required, opened a new log
I20260620 23:01:36.975471 24019 tablet_bootstrap.cc:492] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: No bootstrap required, opened a new log
I20260620 23:01:36.975515 24020 ts_tablet_manager.cc:1397] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:36.975545 24019 ts_tablet_manager.cc:1397] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:36.975687 24020 raft_consensus.cc:348] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:36.975749 24020 raft_consensus.cc:374] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:36.975764 24020 raft_consensus.cc:729] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ce103b7688479c822045a26e935121, State: Initialized, Role: FOLLOWER
I20260620 23:01:36.975718 24019 raft_consensus.cc:348] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:36.975829 24019 raft_consensus.cc:374] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:36.975817 24020 consensus_queue.cc:260] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:36.975863 24019 raft_consensus.cc:729] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd91de572134a018ee2c46763a3a889, State: Initialized, Role: FOLLOWER
I20260620 23:01:36.975931 24020 ts_tablet_manager.cc:1428] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.975926 24019 consensus_queue.cc:260] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:36.975982 24024 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:36.976037 24019 ts_tablet_manager.cc:1428] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:36.976060 24024 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:36.976111 24026 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:36.976203 24026 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 23:01:36.993824 23996 tablet.cc:2369] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:01:37.155833 24026 raft_consensus.cc:482] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:37.155948 24026 raft_consensus.cc:504] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.156527 24026 leader_election.cc:288] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51ce103b7688479c822045a26e935121 (127.21.66.65:34383), 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717)
I20260620 23:01:37.159983 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.156849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:34667 (local address 127.21.66.66:41717)
0620 23:01:37.156909 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.156911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.156924 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:01:37.156956 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.156958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.156969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.157031 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:37.157247 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.157744 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.158949 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.159542 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.159619 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.159628 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.159876 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.159919 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 23:01:37.159991 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.156742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:41717 (local address 127.21.66.67:34667)
0620 23:01:37.156794 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.156807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.156858 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.157054 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.157055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.157056 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:37.157161 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.157163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.157821 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.157822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.158835 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.158838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.159644 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.159645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.159778 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.159783 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.159850 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.159891 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:01:37.160413 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7d28bf09e64c00995d6dd7303ec096" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff577bb049445fe957434f4da166586" is_pre_election: true
I20260620 23:01:37.160570 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.156850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:53403 (local address 127.21.66.65:34383)
0620 23:01:37.156951 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.156953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.157314 (+ 361us) server_negotiation.cc:407] Connection header received
0620 23:01:37.157344 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.157345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.157354 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.157418 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:37.157597 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.158315 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.159552 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.160114 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.160280 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.160291 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.160467 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.160507 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:01:37.160576 23809 raft_consensus.cc:2454] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 0.
I20260620 23:01:37.160591 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.156742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34383 (local address 127.21.66.67:53403)
0620 23:01:37.157108 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.157119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.157182 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.157436 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.157437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.157439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:37.157516 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.157517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.158426 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.158427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.159433 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.159436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.160278 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.160280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.160372 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.160377 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.160456 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.160490 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":131,"thread_start_us":123,"threads_started":1}
I20260620 23:01:37.160789 23881 leader_election.cc:302] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff577bb049445fe957434f4da166586, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:37.160940 23669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7d28bf09e64c00995d6dd7303ec096" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121" is_pre_election: true
I20260620 23:01:37.160969 24026 raft_consensus.cc:2781] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:37.161032 23669 raft_consensus.cc:2454] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 0.
I20260620 23:01:37.161043 24026 raft_consensus.cc:482] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:37.161074 24026 raft_consensus.cc:3037] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:37.161876 24026 raft_consensus.cc:504] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.162055 24026 leader_election.cc:288] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 election: Requested vote from peers 51ce103b7688479c822045a26e935121 (127.21.66.65:34383), 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717)
I20260620 23:01:37.162338 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7d28bf09e64c00995d6dd7303ec096" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff577bb049445fe957434f4da166586"
I20260620 23:01:37.162456 23809 raft_consensus.cc:3037] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:37.162369 23669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7d28bf09e64c00995d6dd7303ec096" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121"
I20260620 23:01:37.162642 23669 raft_consensus.cc:3037] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:37.163218 23809 raft_consensus.cc:2454] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 1.
I20260620 23:01:37.163405 23669 raft_consensus.cc:2454] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 1.
I20260620 23:01:37.163493 23881 leader_election.cc:302] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff577bb049445fe957434f4da166586, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:37.163589 24026 raft_consensus.cc:2781] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:37.163743 24026 raft_consensus.cc:686] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd91de572134a018ee2c46763a3a889, State: Running, Role: LEADER
I20260620 23:01:37.163870 24026 consensus_queue.cc:237] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.164433 24040 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: TxnStatusTablet state changed. Reason: New leader 7fd91de572134a018ee2c46763a3a889. Latest consensus state: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:37.164548 24040 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:01:37.164760 24042 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:01:37.164958 24042 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:01:37.164894 23512 catalog_manager.cc:5495] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd91de572134a018ee2c46763a3a889 (127.21.66.67). New cstate: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } health_report { overall_health: HEALTHY } } }
I20260620 23:01:37.207577 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.204994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.21.66.67:41797)
0620 23:01:37.205059 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.205061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.205070 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:37.205108 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.205109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.205130 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.205215 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:37.205346 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.205824 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.206719 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.206926 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.207008 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.207207 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.207321 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.207419 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.207467 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.207489 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:01:37.207578 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.204884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41797 (local address 127.0.0.1:42752)
0620 23:01:37.204970 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.204982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.205033 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.205222 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.205224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.205225 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:37.205279 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.205282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.205953 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.205955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.206626 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.206628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.207007 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.207008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.207076 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.207096 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.207400 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.207411 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.207452 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.207507 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 23:01:37.209671 23669 raft_consensus.cc:1261] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Refusing update from remote peer 7fd91de572134a018ee2c46763a3a889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:37.209671 23809 raft_consensus.cc:1261] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Refusing update from remote peer 7fd91de572134a018ee2c46763a3a889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:37.210070 24040 consensus_queue.cc:1035] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:37.210242 24026 consensus_queue.cc:1035] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:37.211331 24025 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: TxnStatusTablet state changed. Reason: New leader 7fd91de572134a018ee2c46763a3a889. Latest consensus state: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:37.211436 24025 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:37.211474 24024 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: TxnStatusTablet state changed. Reason: New leader 7fd91de572134a018ee2c46763a3a889. Latest consensus state: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:37.211563 24024 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:37.212371 24040 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:37.212468 24040 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:01:37.212466 24024 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:37.212518 24040 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:37.212549 24024 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 51ce103b7688479c822045a26e935121: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:37.212554 24040 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 7fd91de572134a018ee2c46763a3a889: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:01:37.212687 24025 tablet_replica.cc:439] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } } }
I20260620 23:01:37.212729 24045 mvcc.cc:205] Tried to move back new op lower bound from 7299057652567019520 to 7299057652383002624. Current Snapshot: MvccSnapshot[applied={T|T < 7299057652567019520}]
I20260620 23:01:37.212769 24025 tablet_replica.cc:442] T bc7d28bf09e64c00995d6dd7303ec096 P 5ff577bb049445fe957434f4da166586: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:01:37.213884 24047 mvcc.cc:205] Tried to move back new op lower bound from 7299057652567019520 to 7299057652383002624. Current Snapshot: MvccSnapshot[applied={T|T < 7299057652567019520}]
I20260620 23:01:37.214110 24046 mvcc.cc:205] Tried to move back new op lower bound from 7299057652567019520 to 7299057652383002624. Current Snapshot: MvccSnapshot[applied={T|T < 7299057652567019520}]
I20260620 23:01:37.217593 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.214789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57392 (local address 127.21.66.66:41717)
0620 23:01:37.214830 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.214832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.214839 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:01:37.214872 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.214873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.214892 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.214960 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:37.215105 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.215860 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.216698 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.217098 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.217169 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.217279 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.217399 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.217478 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.217497 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.217518 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:37.222955 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.220268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:46617 (local address 127.21.66.67:41797)
0620 23:01:37.220310 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.220312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.220441 (+ 129us) server_negotiation.cc:407] Connection header received
0620 23:01:37.220521 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.220522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.220532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.220609 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:37.220759 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.221258 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.222039 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.222294 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.222376 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.222569 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.222602 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.222678 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.222846 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.222868 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:01:37.226460 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.220199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41797 (local address 127.21.66.66:46617)
0620 23:01:37.220359 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.220369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.220458 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.220614 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.220615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.220617 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:37.220684 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.220685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.221361 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.221362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.221953 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.221955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.222378 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.222379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.222460 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.222474 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.222778 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.222780 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.222841 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.226348 (+ 3507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260620 23:01:37.230373 23669 raft_consensus.cc:1261] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Refusing update from remote peer 5ff577bb049445fe957434f4da166586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:37.230552 23949 raft_consensus.cc:1261] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Refusing update from remote peer 5ff577bb049445fe957434f4da166586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:37.230751 24025 consensus_queue.cc:1035] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:37.231288 24025 consensus_queue.cc:1035] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:37.241736 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.238931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:41717 (local address 127.21.66.67:32955)
0620 23:01:37.239009 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.239021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.239059 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.239267 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.239268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.239270 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:37.239332 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.239334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.239954 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.239955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.240819 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.240821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.241284 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.241285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.241345 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.241359 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.241599 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.241601 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.241636 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.241677 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:01:37.241751 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.239018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:32955 (local address 127.21.66.66:41717)
0620 23:01:37.239056 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.239058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.239068 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:37.239148 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.239149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.239162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.239217 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:37.239398 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.239868 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.240924 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.241195 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.241280 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.241489 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.241527 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.241586 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.241640 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.241664 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:01:37.252532 23512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:37.252669 23512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:37.253602 23512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Sending DeleteTablet for 3 replicas of tablet b147ce35eae040e58fc6fd06333dda99
I20260620 23:01:37.253983 23929 tablet_service.cc:1514] Processing DeleteTablet for tablet b147ce35eae040e58fc6fd06333dda99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:37 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:42744
I20260620 23:01:37.254000 23649 tablet_service.cc:1514] Processing DeleteTablet for tablet b147ce35eae040e58fc6fd06333dda99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:37 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:50072
I20260620 23:01:37.254482 24065 tablet_replica.cc:330] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: stopping tablet replica
I20260620 23:01:37.254619 23789 tablet_service.cc:1514] Processing DeleteTablet for tablet b147ce35eae040e58fc6fd06333dda99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:37 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 23:01:37.254686 24065 raft_consensus.cc:2229] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:37.254765 24065 raft_consensus.cc:2258] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:37.255066 23512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:37.255239 23512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:37.255244 24066 tablet_replica.cc:330] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: stopping tablet replica
I20260620 23:01:37.255357 24066 raft_consensus.cc:2229] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:37.255481 24066 raft_consensus.cc:2258] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:37.255901 24064 tablet_replica.cc:330] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: stopping tablet replica
I20260620 23:01:37.255999 24066 ts_tablet_manager.cc:1905] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:37.256042 24064 raft_consensus.cc:2229] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:37.256143 24064 raft_consensus.cc:2258] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:37.256843 24064 ts_tablet_manager.cc:1905] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:37.257007 23789 tablet_service.cc:1467] Processing CreateTablet for tablet e7ebebeb511b4581bec18e0b589a5df7 (DEFAULT_TABLE table=test-table [id=3cc89809ea8541dfb5249d4cfde3b6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:37.257131 23789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ebebeb511b4581bec18e0b589a5df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:37.258031 24066 ts_tablet_manager.cc:1918] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:37.258100 24066 log.cc:1196] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/b147ce35eae040e58fc6fd06333dda99
I20260620 23:01:37.258424 24066 ts_tablet_manager.cc:1939] T b147ce35eae040e58fc6fd06333dda99 P 5ff577bb049445fe957434f4da166586: Deleting consensus metadata
I20260620 23:01:37.258601 24018 tablet_bootstrap.cc:492] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: Bootstrap starting.
I20260620 23:01:37.258741 24064 ts_tablet_manager.cc:1918] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:37.258805 24064 log.cc:1196] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/b147ce35eae040e58fc6fd06333dda99
I20260620 23:01:37.259042 24018 tablet_bootstrap.cc:654] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:37.259053 24064 ts_tablet_manager.cc:1939] T b147ce35eae040e58fc6fd06333dda99 P 51ce103b7688479c822045a26e935121: Deleting consensus metadata
I20260620 23:01:37.259661 23498 catalog_manager.cc:4841] TS 51ce103b7688479c822045a26e935121 (127.21.66.65:34383): tablet b147ce35eae040e58fc6fd06333dda99 (table test-table [id=a4478f2f16dd4554860a6430686235b2]) successfully deleted
I20260620 23:01:37.259734 23649 tablet_service.cc:1467] Processing CreateTablet for tablet e7ebebeb511b4581bec18e0b589a5df7 (DEFAULT_TABLE table=test-table [id=3cc89809ea8541dfb5249d4cfde3b6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:37.259866 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ebebeb511b4581bec18e0b589a5df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:37.259899 23929 tablet_service.cc:1467] Processing CreateTablet for tablet e7ebebeb511b4581bec18e0b589a5df7 (DEFAULT_TABLE table=test-table [id=3cc89809ea8541dfb5249d4cfde3b6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:37.260012 23929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ebebeb511b4581bec18e0b589a5df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:37.261071 24020 tablet_bootstrap.cc:492] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: Bootstrap starting.
I20260620 23:01:37.261432 24019 tablet_bootstrap.cc:492] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: Bootstrap starting.
I20260620 23:01:37.261499 24020 tablet_bootstrap.cc:654] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:37.261539 23497 catalog_manager.cc:4841] TS 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717): tablet b147ce35eae040e58fc6fd06333dda99 (table test-table [id=a4478f2f16dd4554860a6430686235b2]) successfully deleted
I20260620 23:01:37.261842 24019 tablet_bootstrap.cc:654] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:37.262480 24019 tablet_bootstrap.cc:492] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: No bootstrap required, opened a new log
I20260620 23:01:37.262540 24019 ts_tablet_manager.cc:1397] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:37.262696 24019 raft_consensus.cc:348] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.262776 24065 ts_tablet_manager.cc:1905] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:37.262789 24019 raft_consensus.cc:374] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:37.263032 24019 raft_consensus.cc:729] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd91de572134a018ee2c46763a3a889, State: Initialized, Role: FOLLOWER
I20260620 23:01:37.263087 24019 consensus_queue.cc:260] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.263192 24019 ts_tablet_manager.cc:1428] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:37.265147 24065 ts_tablet_manager.cc:1918] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:37.265239 24065 log.cc:1196] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/b147ce35eae040e58fc6fd06333dda99
I20260620 23:01:37.265354 24018 tablet_bootstrap.cc:492] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: No bootstrap required, opened a new log
I20260620 23:01:37.265417 24018 ts_tablet_manager.cc:1397] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 23:01:37.265525 24065 ts_tablet_manager.cc:1939] T b147ce35eae040e58fc6fd06333dda99 P 7fd91de572134a018ee2c46763a3a889: Deleting consensus metadata
I20260620 23:01:37.265573 24018 raft_consensus.cc:348] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.265657 24018 raft_consensus.cc:374] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:37.265678 24018 raft_consensus.cc:729] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff577bb049445fe957434f4da166586, State: Initialized, Role: FOLLOWER
I20260620 23:01:37.265725 24018 consensus_queue.cc:260] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.265837 24018 ts_tablet_manager.cc:1428] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:01:37.266304 23498 catalog_manager.cc:4841] TS 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797): tablet b147ce35eae040e58fc6fd06333dda99 (table test-table [id=a4478f2f16dd4554860a6430686235b2]) successfully deleted
I20260620 23:01:37.267042 24020 tablet_bootstrap.cc:492] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: No bootstrap required, opened a new log
I20260620 23:01:37.267103 24020 ts_tablet_manager.cc:1397] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 23:01:37.267261 24020 raft_consensus.cc:348] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.267335 24020 raft_consensus.cc:374] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:37.267357 24020 raft_consensus.cc:729] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ce103b7688479c822045a26e935121, State: Initialized, Role: FOLLOWER
I20260620 23:01:37.267409 24020 consensus_queue.cc:260] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.267498 24020 ts_tablet_manager.cc:1428] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:37.563452 24024 raft_consensus.cc:482] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:37.563563 24024 raft_consensus.cc:504] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.563967 24024 leader_election.cc:288] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717), 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797)
I20260620 23:01:37.568514 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.564290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:36385 (local address 127.21.66.67:41797)
0620 23:01:37.564337 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.564338 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.564445 (+ 107us) server_negotiation.cc:407] Connection header received
0620 23:01:37.564562 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.564563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.564573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.564656 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:37.564972 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.565697 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.567557 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.568128 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.568211 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.568219 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.568407 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.568444 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:37.568514 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.564275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:35211 (local address 127.21.66.66:41717)
0620 23:01:37.564328 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.564330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.564536 (+ 206us) server_negotiation.cc:407] Connection header received
0620 23:01:37.564569 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.564571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.564580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.564661 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:37.564942 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.565700 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.567557 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.568127 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.568211 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.568219 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.568406 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.568441 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:01:37.568535 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.564198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41797 (local address 127.21.66.65:36385)
0620 23:01:37.564342 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.564353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.564465 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.564707 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.564708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.564711 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:37.564825 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.564827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.565821 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.565821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.567441 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.567444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.568231 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.568232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.568312 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.568318 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.568386 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.568436 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":121,"thread_start_us":66,"threads_started":1}
I20260620 23:01:37.568872 24070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.564198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:41717 (local address 127.21.66.65:35211)
0620 23:01:37.564463 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.564473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.564525 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.564707 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.564708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.564713 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:37.564831 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.564832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.565796 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.565798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.567441 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.567444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.568232 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.568233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.568312 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.568316 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.568467 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.568504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":200,"thread_start_us":59,"threads_started":1}
I20260620 23:01:37.568852 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ebebeb511b4581bec18e0b589a5df7" candidate_uuid: "51ce103b7688479c822045a26e935121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd91de572134a018ee2c46763a3a889" is_pre_election: true
I20260620 23:01:37.568982 23949 raft_consensus.cc:2454] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ce103b7688479c822045a26e935121 in term 0.
I20260620 23:01:37.569154 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ebebeb511b4581bec18e0b589a5df7" candidate_uuid: "51ce103b7688479c822045a26e935121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff577bb049445fe957434f4da166586" is_pre_election: true
I20260620 23:01:37.569236 23809 raft_consensus.cc:2454] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ce103b7688479c822045a26e935121 in term 0.
I20260620 23:01:37.569221 23602 leader_election.cc:302] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce103b7688479c822045a26e935121, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:37.569386 24024 raft_consensus.cc:2781] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:37.569422 24024 raft_consensus.cc:482] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:37.569442 24024 raft_consensus.cc:3037] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:37.570194 24024 raft_consensus.cc:504] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.570344 24024 leader_election.cc:288] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [CANDIDATE]: Term 1 election: Requested vote from peers 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717), 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797)
I20260620 23:01:37.570493 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ebebeb511b4581bec18e0b589a5df7" candidate_uuid: "51ce103b7688479c822045a26e935121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff577bb049445fe957434f4da166586"
I20260620 23:01:37.570580 23809 raft_consensus.cc:3037] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:37.570518 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ebebeb511b4581bec18e0b589a5df7" candidate_uuid: "51ce103b7688479c822045a26e935121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd91de572134a018ee2c46763a3a889"
I20260620 23:01:37.570619 23949 raft_consensus.cc:3037] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:37.571295 23949 raft_consensus.cc:2454] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ce103b7688479c822045a26e935121 in term 1.
I20260620 23:01:37.571378 23809 raft_consensus.cc:2454] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51ce103b7688479c822045a26e935121 in term 1.
I20260620 23:01:37.571512 23602 leader_election.cc:302] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce103b7688479c822045a26e935121, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:37.571604 24024 raft_consensus.cc:2781] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:37.571768 24024 raft_consensus.cc:686] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 1 LEADER]: Becoming Leader. State: Replica: 51ce103b7688479c822045a26e935121, State: Running, Role: LEADER
I20260620 23:01:37.571852 24024 consensus_queue.cc:237] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:37.572777 23512 catalog_manager.cc:5495] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51ce103b7688479c822045a26e935121 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "51ce103b7688479c822045a26e935121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:37.625355 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.622943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.21.66.65:34383)
0620 23:01:37.623024 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.623026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.623043 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:01:37.623086 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.623087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.623112 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.623204 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:37.623313 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.623797 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.624532 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.624742 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.624836 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.624993 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.625109 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.625198 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.625242 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.625266 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:01:37.630882 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.628189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41797 (local address 127.21.66.65:46881)
0620 23:01:37.628405 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.628417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.628484 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.628633 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.628633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.628635 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:37.628707 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.628709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.629340 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.629341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.629968 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.629970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.630415 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.630417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.630479 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.630502 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.630696 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.630698 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.630741 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.630787 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260620 23:01:37.630860 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.628289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:46881 (local address 127.21.66.67:41797)
0620 23:01:37.628364 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.628366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.628487 (+ 121us) server_negotiation.cc:407] Connection header received
0620 23:01:37.628541 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.628542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.628557 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.628630 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:37.628788 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.629252 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.630067 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.630307 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.630395 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.630599 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.630634 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.630693 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.630754 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.630777 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:01:37.633253 23809 raft_consensus.cc:1261] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Refusing update from remote peer 51ce103b7688479c822045a26e935121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:37.633286 23949 raft_consensus.cc:1261] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Refusing update from remote peer 51ce103b7688479c822045a26e935121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:37.633546 24072 consensus_queue.cc:1035] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:37.633698 24024 consensus_queue.cc:1035] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:37.641062 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.638537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:34655 (local address 127.21.66.65:34383)
0620 23:01:37.638576 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:37.638578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:37.638587 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:37.638652 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:37.638653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:37.638663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:37.638728 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:37.638843 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.639440 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.640195 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.640585 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.640653 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.640828 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.640864 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.640914 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:37.640989 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 23:01:37.641007 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:01:37.641083 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:37.638454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34383 (local address 127.21.66.67:34655)
0620 23:01:37.638545 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:37.638557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:37.638597 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:37.638731 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:37.638732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:37.638733 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:37.638782 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.638784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.639535 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.639536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.640108 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:37.640110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:37.640659 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:37.640660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:37.640750 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:37.640756 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:37.640934 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:37.640936 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:01:37.640980 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 23:01:37.641023 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 23:01:37.641530 24047 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:01:39.162290 23511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:39.162410 23511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:39.163417 23511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Sending DeleteTablet for 3 replicas of tablet e7ebebeb511b4581bec18e0b589a5df7
I20260620 23:01:39.163779 23789 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ebebeb511b4581bec18e0b589a5df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:39 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 23:01:39.163780 23649 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ebebeb511b4581bec18e0b589a5df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:39 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:50072
I20260620 23:01:39.163782 23928 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ebebeb511b4581bec18e0b589a5df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:39 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:42744
I20260620 23:01:39.164011 24105 tablet_replica.cc:330] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: stopping tablet replica
I20260620 23:01:39.164083 24105 raft_consensus.cc:2229] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:39.164163 24105 raft_consensus.cc:2258] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:39.164309 24103 tablet_replica.cc:330] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: stopping tablet replica
I20260620 23:01:39.164436 24104 tablet_replica.cc:330] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: stopping tablet replica
I20260620 23:01:39.164461 24105 ts_tablet_manager.cc:1905] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:39.164506 24104 raft_consensus.cc:2229] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:39.164578 24103 raft_consensus.cc:2229] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:39.164697 24103 raft_consensus.cc:2258] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:39.164783 24104 raft_consensus.cc:2258] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:39.165063 24104 ts_tablet_manager.cc:1905] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:39.166579 24105 ts_tablet_manager.cc:1918] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:01:39.166716 24105 log.cc:1196] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/e7ebebeb511b4581bec18e0b589a5df7
I20260620 23:01:39.166863 24104 ts_tablet_manager.cc:1918] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:01:39.166930 24104 log.cc:1196] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/e7ebebeb511b4581bec18e0b589a5df7
I20260620 23:01:39.167052 24105 ts_tablet_manager.cc:1939] T e7ebebeb511b4581bec18e0b589a5df7 P 5ff577bb049445fe957434f4da166586: Deleting consensus metadata
I20260620 23:01:39.167141 24104 ts_tablet_manager.cc:1939] T e7ebebeb511b4581bec18e0b589a5df7 P 7fd91de572134a018ee2c46763a3a889: Deleting consensus metadata
I20260620 23:01:39.167392 24103 ts_tablet_manager.cc:1905] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:39.167483 23497 catalog_manager.cc:4841] TS 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717): tablet e7ebebeb511b4581bec18e0b589a5df7 (table test-table [id=3cc89809ea8541dfb5249d4cfde3b6cc]) successfully deleted
I20260620 23:01:39.168848 24103 ts_tablet_manager.cc:1918] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:01:39.168919 24103 log.cc:1196] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/e7ebebeb511b4581bec18e0b589a5df7
I20260620 23:01:39.169189 24103 ts_tablet_manager.cc:1939] T e7ebebeb511b4581bec18e0b589a5df7 P 51ce103b7688479c822045a26e935121: Deleting consensus metadata
I20260620 23:01:39.169579 23498 catalog_manager.cc:4841] TS 51ce103b7688479c822045a26e935121 (127.21.66.65:34383): tablet e7ebebeb511b4581bec18e0b589a5df7 (table test-table [id=3cc89809ea8541dfb5249d4cfde3b6cc]) successfully deleted
I20260620 23:01:39.170179 23498 catalog_manager.cc:4841] TS 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797): tablet e7ebebeb511b4581bec18e0b589a5df7 (table test-table [id=3cc89809ea8541dfb5249d4cfde3b6cc]) successfully deleted
Jun 20 23:01:39 dist-test-slave-qgtk krb5kdc[23467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:01:39 dist-test-slave-qgtk krb5kdc[23467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:39.196493 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:39.182563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.21.66.126:39291)
0620 23:01:39.182715 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:39.182717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:39.182832 (+ 115us) server_negotiation.cc:407] Connection header received
0620 23:01:39.182869 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:39.182869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:39.182881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:39.182964 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:39.183670 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.184219 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.185076 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.185291 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.185352 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:39.193024 (+ 7672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:39.193027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:39.193027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:39.193037 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:39.194982 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:39.195539 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:39.195540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:39.195541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:39.195610 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:39.195869 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:39.195869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:39.195869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:39.196005 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:39.196099 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:39.196240 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:01:39.196326 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260620 23:01:39.198515 23511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:39.198709 23511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:39.200212 23649 tablet_service.cc:1467] Processing CreateTablet for tablet b04d799336ee4867b3d9e107d42b5bdc (DEFAULT_TABLE table=test-table [id=4631fbd4ce6042aa925ef15e1c6aa41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.200284 23789 tablet_service.cc:1467] Processing CreateTablet for tablet b04d799336ee4867b3d9e107d42b5bdc (DEFAULT_TABLE table=test-table [id=4631fbd4ce6042aa925ef15e1c6aa41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.200343 23928 tablet_service.cc:1467] Processing CreateTablet for tablet b04d799336ee4867b3d9e107d42b5bdc (DEFAULT_TABLE table=test-table [id=4631fbd4ce6042aa925ef15e1c6aa41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.200381 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04d799336ee4867b3d9e107d42b5bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.200416 23789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04d799336ee4867b3d9e107d42b5bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.200457 23928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04d799336ee4867b3d9e107d42b5bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.201534 24114 tablet_bootstrap.cc:492] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: Bootstrap starting.
I20260620 23:01:39.201938 24115 tablet_bootstrap.cc:492] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: Bootstrap starting.
I20260620 23:01:39.202047 24114 tablet_bootstrap.cc:654] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.202222 24116 tablet_bootstrap.cc:492] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: Bootstrap starting.
I20260620 23:01:39.202445 24115 tablet_bootstrap.cc:654] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.202622 24116 tablet_bootstrap.cc:654] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.202896 24114 tablet_bootstrap.cc:492] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: No bootstrap required, opened a new log
I20260620 23:01:39.202952 24114 ts_tablet_manager.cc:1397] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:39.203037 24115 tablet_bootstrap.cc:492] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: No bootstrap required, opened a new log
I20260620 23:01:39.203086 24115 ts_tablet_manager.cc:1397] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:39.203068 24114 raft_consensus.cc:348] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.203167 24114 raft_consensus.cc:374] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.203197 24114 raft_consensus.cc:729] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff577bb049445fe957434f4da166586, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.203212 24115 raft_consensus.cc:348] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.203284 24115 raft_consensus.cc:374] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.203274 24114 consensus_queue.cc:260] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.203312 24115 raft_consensus.cc:729] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ce103b7688479c822045a26e935121, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.203383 24115 consensus_queue.cc:260] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.203461 24114 ts_tablet_manager.cc:1428] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 23:01:39.203504 24115 ts_tablet_manager.cc:1428] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:39.204033 24116 tablet_bootstrap.cc:492] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: No bootstrap required, opened a new log
I20260620 23:01:39.204094 24116 ts_tablet_manager.cc:1397] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:39.204198 24116 raft_consensus.cc:348] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.204280 24116 raft_consensus.cc:374] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.204312 24116 raft_consensus.cc:729] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd91de572134a018ee2c46763a3a889, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.204356 24116 consensus_queue.cc:260] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.204452 24116 ts_tablet_manager.cc:1428] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:39.368902 24120 raft_consensus.cc:482] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:39.369136 24120 raft_consensus.cc:504] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.369349 24120 leader_election.cc:288] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51ce103b7688479c822045a26e935121 (127.21.66.65:34383), 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797)
I20260620 23:01:39.369692 23669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04d799336ee4867b3d9e107d42b5bdc" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121" is_pre_election: true
I20260620 23:01:39.369692 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04d799336ee4867b3d9e107d42b5bdc" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd91de572134a018ee2c46763a3a889" is_pre_election: true
I20260620 23:01:39.369810 23949 raft_consensus.cc:2454] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 0.
I20260620 23:01:39.369810 23669 raft_consensus.cc:2454] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 0.
I20260620 23:01:39.369982 23742 leader_election.cc:302] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff577bb049445fe957434f4da166586, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:39.370078 24120 raft_consensus.cc:2781] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:39.370150 24120 raft_consensus.cc:482] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:39.370182 24120 raft_consensus.cc:3037] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.370939 24120 raft_consensus.cc:504] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.371076 24120 leader_election.cc:288] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 election: Requested vote from peers 51ce103b7688479c822045a26e935121 (127.21.66.65:34383), 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797)
I20260620 23:01:39.371268 23669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04d799336ee4867b3d9e107d42b5bdc" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121"
I20260620 23:01:39.371383 23669 raft_consensus.cc:3037] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.371343 23949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04d799336ee4867b3d9e107d42b5bdc" candidate_uuid: "5ff577bb049445fe957434f4da166586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fd91de572134a018ee2c46763a3a889"
I20260620 23:01:39.371474 23949 raft_consensus.cc:3037] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.372058 23669 raft_consensus.cc:2454] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 1.
I20260620 23:01:39.372210 23949 raft_consensus.cc:2454] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff577bb049445fe957434f4da166586 in term 1.
I20260620 23:01:39.372409 23742 leader_election.cc:302] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce103b7688479c822045a26e935121, 5ff577bb049445fe957434f4da166586; no voters:
I20260620 23:01:39.372525 24120 raft_consensus.cc:2781] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:39.372793 24120 raft_consensus.cc:686] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff577bb049445fe957434f4da166586, State: Running, Role: LEADER
I20260620 23:01:39.372923 24120 consensus_queue.cc:237] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.373685 23512 catalog_manager.cc:5495] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff577bb049445fe957434f4da166586 (127.21.66.66). New cstate: current_term: 1 leader_uuid: "5ff577bb049445fe957434f4da166586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:39.392071 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:39.389139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.21.66.66:41717)
0620 23:01:39.389311 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:39.389313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:39.389320 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:01:39.389352 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:39.389352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:39.389362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:39.389446 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:39.389600 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.390318 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.391215 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.391458 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.391545 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:39.391751 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:39.391789 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:39.391867 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:39.391937 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:01:39.391965 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260620 23:01:39.395124 23949 raft_consensus.cc:1261] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Refusing update from remote peer 5ff577bb049445fe957434f4da166586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:39.395198 23669 raft_consensus.cc:1261] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Refusing update from remote peer 5ff577bb049445fe957434f4da166586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:39.395429 24120 consensus_queue.cc:1035] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:39.395548 24121 consensus_queue.cc:1035] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:39.407335 23512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:39.407449 23512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:39.408262 23512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Sending DeleteTablet for 3 replicas of tablet b04d799336ee4867b3d9e107d42b5bdc
I20260620 23:01:39.408545 23649 tablet_service.cc:1514] Processing DeleteTablet for tablet b04d799336ee4867b3d9e107d42b5bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:39 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:50072
I20260620 23:01:39.408659 24103 tablet_replica.cc:330] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: stopping tablet replica
I20260620 23:01:39.408635 23789 tablet_service.cc:1514] Processing DeleteTablet for tablet b04d799336ee4867b3d9e107d42b5bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:39 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 23:01:39.408638 23928 tablet_service.cc:1514] Processing DeleteTablet for tablet b04d799336ee4867b3d9e107d42b5bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:39 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:42744
I20260620 23:01:39.408704 24103 raft_consensus.cc:2229] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:39.408742 24105 tablet_replica.cc:330] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: stopping tablet replica
I20260620 23:01:39.408792 24104 tablet_replica.cc:330] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: stopping tablet replica
I20260620 23:01:39.408849 24105 raft_consensus.cc:2229] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:39.408874 24104 raft_consensus.cc:2229] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:39.408741 24103 raft_consensus.cc:2258] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:39.408917 24104 raft_consensus.cc:2258] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:39.408946 24105 raft_consensus.cc:2258] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:39.409242 24105 ts_tablet_manager.cc:1905] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:39.409214 23512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:39.409358 23512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:39.409348 24103 ts_tablet_manager.cc:1905] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:39.411113 24104 ts_tablet_manager.cc:1905] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:39.411142 23928 tablet_service.cc:1467] Processing CreateTablet for tablet c929b87841924a9bbf7287904fb38291 (DEFAULT_TABLE table=test-table [id=5be7348cbd0a45af8d785bcfccd5eeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.411145 23789 tablet_service.cc:1467] Processing CreateTablet for tablet c929b87841924a9bbf7287904fb38291 (DEFAULT_TABLE table=test-table [id=5be7348cbd0a45af8d785bcfccd5eeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.411238 23928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c929b87841924a9bbf7287904fb38291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.412566 24104 ts_tablet_manager.cc:1918] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:39.412588 24116 tablet_bootstrap.cc:492] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: Bootstrap starting.
I20260620 23:01:39.412636 24104 log.cc:1196] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/b04d799336ee4867b3d9e107d42b5bdc
I20260620 23:01:39.412873 24104 ts_tablet_manager.cc:1939] T b04d799336ee4867b3d9e107d42b5bdc P 7fd91de572134a018ee2c46763a3a889: Deleting consensus metadata
I20260620 23:01:39.413013 24116 tablet_bootstrap.cc:654] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.413045 23649 tablet_service.cc:1467] Processing CreateTablet for tablet c929b87841924a9bbf7287904fb38291 (DEFAULT_TABLE table=test-table [id=5be7348cbd0a45af8d785bcfccd5eeb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:39.413259 23789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c929b87841924a9bbf7287904fb38291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.413357 24103 ts_tablet_manager.cc:1918] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:39.413411 24103 log.cc:1196] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/b04d799336ee4867b3d9e107d42b5bdc
I20260620 23:01:39.413601 24103 ts_tablet_manager.cc:1939] T b04d799336ee4867b3d9e107d42b5bdc P 51ce103b7688479c822045a26e935121: Deleting consensus metadata
I20260620 23:01:39.413595 24116 tablet_bootstrap.cc:492] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: No bootstrap required, opened a new log
I20260620 23:01:39.413640 24116 ts_tablet_manager.cc:1397] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:39.413729 23649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c929b87841924a9bbf7287904fb38291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:39.413753 24116 raft_consensus.cc:348] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.413816 24116 raft_consensus.cc:374] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.413839 24116 raft_consensus.cc:729] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd91de572134a018ee2c46763a3a889, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.413888 24116 consensus_queue.cc:260] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.413982 24116 ts_tablet_manager.cc:1428] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:01:39.414481 23498 catalog_manager.cc:4841] TS 51ce103b7688479c822045a26e935121 (127.21.66.65:34383): tablet b04d799336ee4867b3d9e107d42b5bdc (table test-table [id=4631fbd4ce6042aa925ef15e1c6aa41f]) successfully deleted
I20260620 23:01:39.414881 24115 tablet_bootstrap.cc:492] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121: Bootstrap starting.
I20260620 23:01:39.413275 24105 ts_tablet_manager.cc:1918] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:01:39.415115 24105 log.cc:1196] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/b04d799336ee4867b3d9e107d42b5bdc
I20260620 23:01:39.415010 24114 tablet_bootstrap.cc:492] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: Bootstrap starting.
I20260620 23:01:39.415205 24115 tablet_bootstrap.cc:654] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.415362 24105 ts_tablet_manager.cc:1939] T b04d799336ee4867b3d9e107d42b5bdc P 5ff577bb049445fe957434f4da166586: Deleting consensus metadata
I20260620 23:01:39.415444 24114 tablet_bootstrap.cc:654] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:39.415709 23498 catalog_manager.cc:4841] TS 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797): tablet b04d799336ee4867b3d9e107d42b5bdc (table test-table [id=4631fbd4ce6042aa925ef15e1c6aa41f]) successfully deleted
I20260620 23:01:39.415773 24115 tablet_bootstrap.cc:492] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121: No bootstrap required, opened a new log
I20260620 23:01:39.415823 24115 ts_tablet_manager.cc:1397] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:39.415978 23497 catalog_manager.cc:4841] TS 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717): tablet b04d799336ee4867b3d9e107d42b5bdc (table test-table [id=4631fbd4ce6042aa925ef15e1c6aa41f]) successfully deleted
I20260620 23:01:39.415967 24115 raft_consensus.cc:348] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.416050 24115 raft_consensus.cc:374] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.416070 24115 raft_consensus.cc:729] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ce103b7688479c822045a26e935121, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.416071 24114 tablet_bootstrap.cc:492] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: No bootstrap required, opened a new log
I20260620 23:01:39.416117 24114 ts_tablet_manager.cc:1397] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:39.416119 24115 consensus_queue.cc:260] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.416237 24115 ts_tablet_manager.cc:1428] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:39.416239 24114 raft_consensus.cc:348] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.416308 24114 raft_consensus.cc:374] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:39.416323 24114 raft_consensus.cc:729] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff577bb049445fe957434f4da166586, State: Initialized, Role: FOLLOWER
I20260620 23:01:39.416355 24114 consensus_queue.cc:260] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.416476 24114 ts_tablet_manager.cc:1428] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:39.577930 24098 raft_consensus.cc:482] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:39.578068 24098 raft_consensus.cc:504] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.578336 24098 leader_election.cc:288] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51ce103b7688479c822045a26e935121 (127.21.66.65:34383), 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717)
I20260620 23:01:39.578585 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c929b87841924a9bbf7287904fb38291" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff577bb049445fe957434f4da166586" is_pre_election: true
I20260620 23:01:39.578608 23668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c929b87841924a9bbf7287904fb38291" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121" is_pre_election: true
I20260620 23:01:39.578686 23809 raft_consensus.cc:2454] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 0.
I20260620 23:01:39.578694 23668 raft_consensus.cc:2454] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 0.
I20260620 23:01:39.578867 23882 leader_election.cc:302] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ce103b7688479c822045a26e935121, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:39.578970 24098 raft_consensus.cc:2781] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:39.579031 24098 raft_consensus.cc:482] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:39.579056 24098 raft_consensus.cc:3037] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.579737 24098 raft_consensus.cc:504] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.579881 24098 leader_election.cc:288] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 election: Requested vote from peers 51ce103b7688479c822045a26e935121 (127.21.66.65:34383), 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717)
I20260620 23:01:39.580039 23809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c929b87841924a9bbf7287904fb38291" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ff577bb049445fe957434f4da166586"
I20260620 23:01:39.580052 23668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c929b87841924a9bbf7287904fb38291" candidate_uuid: "7fd91de572134a018ee2c46763a3a889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ce103b7688479c822045a26e935121"
I20260620 23:01:39.580121 23809 raft_consensus.cc:3037] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.580133 23668 raft_consensus.cc:3037] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:39.580852 23809 raft_consensus.cc:2454] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 1.
I20260620 23:01:39.581012 23668 raft_consensus.cc:2454] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fd91de572134a018ee2c46763a3a889 in term 1.
I20260620 23:01:39.581048 23881 leader_election.cc:302] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff577bb049445fe957434f4da166586, 7fd91de572134a018ee2c46763a3a889; no voters:
I20260620 23:01:39.581161 24098 raft_consensus.cc:2781] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:39.581208 24098 raft_consensus.cc:686] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd91de572134a018ee2c46763a3a889, State: Running, Role: LEADER
I20260620 23:01:39.581259 24098 consensus_queue.cc:237] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } }
I20260620 23:01:39.581883 23512 catalog_manager.cc:5495] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fd91de572134a018ee2c46763a3a889 (127.21.66.67). New cstate: current_term: 1 leader_uuid: "7fd91de572134a018ee2c46763a3a889" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fd91de572134a018ee2c46763a3a889" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41797 } health_report { overall_health: HEALTHY } } }
I20260620 23:01:39.603861 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:39.601258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.21.66.67:41797)
0620 23:01:39.601453 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:39.601456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:39.601464 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:39.601489 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:39.601489 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:39.601501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:39.601592 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:39.601716 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.602276 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.603068 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.603295 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.603388 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:39.603557 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:39.603598 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:39.603668 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:39.603726 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 23:01:39.603749 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
I20260620 23:01:39.608676 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:39.606449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:59445 (local address 127.21.66.67:41797)
0620 23:01:39.606485 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:39.606487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:39.606496 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:39.606518 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:39.606519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:39.606526 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:39.606571 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:39.606649 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.607098 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.608008 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.608185 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.608242 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:39.608462 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:39.608486 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:39.608523 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:39.608578 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:01:39.608595 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:01:39.608767 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:39.606258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41797 (local address 127.21.66.67:59445)
0620 23:01:39.606394 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:39.606414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:39.606457 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:39.606573 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:39.606573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:39.606574 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:01:39.606613 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:39.606614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.607183 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.607184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:39.607921 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:39.607923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:39.608281 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:39.608282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:39.608387 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:39.608392 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:01:39.608540 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:01:39.608542 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:01:39.608577 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 23:01:39.608618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":93,"threads_started":1}
I20260620 23:01:39.610889 23668 raft_consensus.cc:1261] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Refusing update from remote peer 7fd91de572134a018ee2c46763a3a889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:39.610884 23809 raft_consensus.cc:1261] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Refusing update from remote peer 7fd91de572134a018ee2c46763a3a889: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:39.611152 24098 consensus_queue.cc:1035] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ff577bb049445fe957434f4da166586" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:39.611351 24098 consensus_queue.cc:1035] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ce103b7688479c822045a26e935121" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:01:39.615523 24082 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:01:41.140340 23510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:41.140499 23510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:41.141638 23510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e160ef6515947699af602da657fa80d: Sending DeleteTablet for 3 replicas of tablet c929b87841924a9bbf7287904fb38291
I20260620 23:01:41.142001 23649 tablet_service.cc:1514] Processing DeleteTablet for tablet c929b87841924a9bbf7287904fb38291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:41 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:50072
I20260620 23:01:41.142023 23789 tablet_service.cc:1514] Processing DeleteTablet for tablet c929b87841924a9bbf7287904fb38291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:41 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 23:01:41.142251 23928 tablet_service.cc:1514] Processing DeleteTablet for tablet c929b87841924a9bbf7287904fb38291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:41 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:42744
I20260620 23:01:41.142313 24159 tablet_replica.cc:330] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121: stopping tablet replica
I20260620 23:01:41.142392 24159 raft_consensus.cc:2229] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:41.142453 24159 raft_consensus.cc:2258] T c929b87841924a9bbf7287904fb38291 P 51ce103b7688479c822045a26e935121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:41.142472 24161 tablet_replica.cc:330] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: stopping tablet replica
I20260620 23:01:41.142555 24161 raft_consensus.cc:2229] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:41.142660 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 23584
I20260620 23:01:41.142678 24161 raft_consensus.cc:2258] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:41.143352 24160 tablet_replica.cc:330] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: stopping tablet replica
I20260620 23:01:41.143517 24160 raft_consensus.cc:2229] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:41.143606 24160 raft_consensus.cc:2258] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:41.143651 24161 ts_tablet_manager.cc:1905] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:41.143879 24160 ts_tablet_manager.cc:1905] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:41.145457 24160 ts_tablet_manager.cc:1918] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:01:41.145551 24160 log.cc:1196] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/c929b87841924a9bbf7287904fb38291
I20260620 23:01:41.145828 24160 ts_tablet_manager.cc:1939] T c929b87841924a9bbf7287904fb38291 P 5ff577bb049445fe957434f4da166586: Deleting consensus metadata
I20260620 23:01:41.145939 24161 ts_tablet_manager.cc:1918] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:01:41.146011 24161 log.cc:1196] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/c929b87841924a9bbf7287904fb38291
I20260620 23:01:41.146281 23497 catalog_manager.cc:4841] TS 5ff577bb049445fe957434f4da166586 (127.21.66.66:41717): tablet c929b87841924a9bbf7287904fb38291 (table test-table [id=5be7348cbd0a45af8d785bcfccd5eeb0]) successfully deleted
I20260620 23:01:41.146333 24161 ts_tablet_manager.cc:1939] T c929b87841924a9bbf7287904fb38291 P 7fd91de572134a018ee2c46763a3a889: Deleting consensus metadata
I20260620 23:01:41.146814 23498 catalog_manager.cc:4841] TS 7fd91de572134a018ee2c46763a3a889 (127.21.66.67:41797): tablet c929b87841924a9bbf7287904fb38291 (table test-table [id=5be7348cbd0a45af8d785bcfccd5eeb0]) successfully deleted
W20260620 23:01:41.151427 23498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:34383 (error 108)
I20260620 23:01:41.151656 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 23724
I20260620 23:01:41.152863 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:41.152608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34383 (local address 127.0.0.1:59494)
0620 23:01:41.152749 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:41.152814 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:34383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
W20260620 23:01:41.153103 23498 catalog_manager.cc:4584] TS 51ce103b7688479c822045a26e935121 (127.21.66.65:34383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c929b87841924a9bbf7287904fb38291: Network error: Client connection negotiation failed: client connection to 127.21.66.65:34383: connect: Connection refused (error 111)
I20260620 23:01:41.158622 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 23864
I20260620 23:01:41.164944 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 23480
2026-06-20T23:01:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6369 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:41 dist-test-slave-qgtk krb5kdc[24166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:41 dist-test-slave-qgtk krb5kdc[24166](info): set up 2 sockets
Jun 20 23:01:41 dist-test-slave-qgtk krb5kdc[24166](info): commencing operation
krb5kdc: starting...
W20260620 23:01:42.266533 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:42 dist-test-slave-qgtk krb5kdc[24166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:42.406766 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46683
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:42.495225 24179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:42.495481 24179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:42.495507 24179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:42.496800 24179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:42.496858 24179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:42.496873 24179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:42.496886 24179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:42.496898 24179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:42.498602 24179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:46683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:42.498899 24179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:42.499184 24179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:42.505414 24185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:42.505419 24184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:42.505440 24187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:42.505483 24179 server_base.cc:1034] running on GCE node
I20260620 23:01:42.506186 24179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:42.506669 24179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:42.507884 24179 hybrid_clock.cc:648] HybridClock initialized: now 1781996502507881 us; error 52 us; skew 500 ppm
Jun 20 23:01:42 dist-test-slave-qgtk krb5kdc[24166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996502, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:42.511039 24179 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:42.511623 24179 webserver.cc:460] Webserver started at http://127.21.66.126:43915/ using document root <none> and password file <none>
I20260620 23:01:42.511909 24179 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:42.511986 24179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:42.512144 24179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:42.513423 24179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "6b1a5067a02c4afea6344b8b06b1f428"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "199679cc70bb232754e8d212a0317d93"
server_key_iv: "38f9cc07e8e1bc8ad064b2bef620a833"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.513820 24179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "6b1a5067a02c4afea6344b8b06b1f428"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "199679cc70bb232754e8d212a0317d93"
server_key_iv: "38f9cc07e8e1bc8ad064b2bef620a833"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.515564 24179 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:01:42.516541 24194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:42.516793 24179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:42.516907 24179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "6b1a5067a02c4afea6344b8b06b1f428"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "199679cc70bb232754e8d212a0317d93"
server_key_iv: "38f9cc07e8e1bc8ad064b2bef620a833"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.517042 24179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:42.553263 24179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:42.555861 24179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:42.556056 24179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:42.560586 24179 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:46683
I20260620 23:01:42.560619 24246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:46683 every 8 connection(s)
I20260620 23:01:42.560998 24179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:42.561982 24247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:42.562857 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 24179
I20260620 23:01:42.562958 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:42.563169 21769 external_mini_cluster.cc:1428] Setting key 33bc53e65a91090d7ec2f8388a1b57b9
I20260620 23:01:42.565352 24247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428: Bootstrap starting.
I20260620 23:01:42.566072 24247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:42.566421 24247 log.cc:824] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:42.567282 24247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428: No bootstrap required, opened a new log
I20260620 23:01:42.570194 24247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a5067a02c4afea6344b8b06b1f428" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46683 } }
I20260620 23:01:42.570384 24247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:42.570413 24247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1a5067a02c4afea6344b8b06b1f428, State: Initialized, Role: FOLLOWER
I20260620 23:01:42.570576 24247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a5067a02c4afea6344b8b06b1f428" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46683 } }
I20260620 23:01:42.570653 24247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:42.570681 24247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:42.570756 24247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:01:42 dist-test-slave-qgtk krb5kdc[24166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:42.571470 24247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a5067a02c4afea6344b8b06b1f428" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46683 } }
I20260620 23:01:42.571565 24247 leader_election.cc:302] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b1a5067a02c4afea6344b8b06b1f428; no voters:
I20260620 23:01:42.571807 24247 leader_election.cc:288] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:42.571844 24252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:42.571962 24252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1a5067a02c4afea6344b8b06b1f428, State: Running, Role: LEADER
I20260620 23:01:42.572063 24252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a5067a02c4afea6344b8b06b1f428" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46683 } }
I20260620 23:01:42.572113 24247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:42.572522 24252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b1a5067a02c4afea6344b8b06b1f428. Latest consensus state: current_term: 1 leader_uuid: "6b1a5067a02c4afea6344b8b06b1f428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a5067a02c4afea6344b8b06b1f428" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46683 } } }
I20260620 23:01:42.572618 24252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:42.572556 24253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b1a5067a02c4afea6344b8b06b1f428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1a5067a02c4afea6344b8b06b1f428" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46683 } } }
I20260620 23:01:42.573128 24253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:42.573737 24261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:42.574509 24261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:42.575987 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:42.563759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.21.66.126:46683)
0620 23:01:42.563937 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:42.563950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:42.563984 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:01:42.564384 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:42.564390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:42.564500 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:42.564626 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:42.565517 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.566570 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.567663 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.568087 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.568172 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:42.571462 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:42.571467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:42.571467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:42.571481 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:42.574023 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:42.574636 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:42.574640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:42.574640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:42.574748 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:42.574986 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:42.574987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:42.574987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:42.575321 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:42.575477 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:42.575635 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:01:42.575785 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 23:01:42.576452 24261 catalog_manager.cc:1347] Generated new cluster ID: 71e2848e88b546c2a339e503c168c631
I20260620 23:01:42.576504 24261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:42.607604 24261 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:42.608377 24261 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:42.614286 24261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b1a5067a02c4afea6344b8b06b1f428: Generated new TSK 0
I20260620 23:01:42.614516 24261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:42.646274 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46683
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:42.734968 24273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:42.735214 24273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:42.735242 24273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:42.736527 24273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:42.736577 24273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:42.736605 24273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:42.738137 24273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:42.738420 24273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:42.738677 24273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:42.744288 24279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:42.744305 24278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:42.744369 24281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:42.744692 24273 server_base.cc:1034] running on GCE node
I20260620 23:01:42.744947 24273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:42.745260 24273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:42.746426 24273 hybrid_clock.cc:648] HybridClock initialized: now 1781996502746443 us; error 63 us; skew 500 ppm
Jun 20 23:01:42 dist-test-slave-qgtk krb5kdc[24166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996502, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:42.749212 24273 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:42.749713 24273 webserver.cc:460] Webserver started at http://127.21.66.65:41055/ using document root <none> and password file <none>
I20260620 23:01:42.749943 24273 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:42.749992 24273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:42.750169 24273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:42.751384 24273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "7da70e11241c4ef6b1dc3255b67b9807"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "ac4a7547e6cbd54bc3f519b2cf77b8d2"
server_key_iv: "7769f192f8e96af8db1b6745d1d409df"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.751727 24273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "7da70e11241c4ef6b1dc3255b67b9807"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "ac4a7547e6cbd54bc3f519b2cf77b8d2"
server_key_iv: "7769f192f8e96af8db1b6745d1d409df"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.753258 24273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:42.754231 24288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:42.754541 24273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:42.754667 24273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "7da70e11241c4ef6b1dc3255b67b9807"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "ac4a7547e6cbd54bc3f519b2cf77b8d2"
server_key_iv: "7769f192f8e96af8db1b6745d1d409df"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.754786 24273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:42.770813 24273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:42.773772 24273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:42.773955 24273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:42.774281 24273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:42.774667 24273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:42.774709 24273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:42.774768 24273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:42.774803 24273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:42.781526 24273 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:35089
I20260620 23:01:42.781610 24401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:35089 every 8 connection(s)
I20260620 23:01:42.781996 24273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:01:42 dist-test-slave-qgtk krb5kdc[24166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996502, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:42.792050 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 24273
I20260620 23:01:42.792174 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:42.792449 21769 external_mini_cluster.cc:1428] Setting key 86605f6dcce1ff61e9df3398e55d92f8
I20260620 23:01:42.793560 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:42.782994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:44937 (local address 127.21.66.126:46683)
0620 23:01:42.783074 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:42.783076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:42.783274 (+ 198us) server_negotiation.cc:407] Connection header received
0620 23:01:42.783803 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:42.783804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:42.783816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:42.783888 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:42.785126 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.785718 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.786888 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.787112 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.787208 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:42.789869 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:42.789870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:42.789871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:42.789879 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:42.791573 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:42.792235 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:42.792237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:42.792237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:42.792321 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:42.792708 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:42.792709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:42.792709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:42.792842 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:42.792917 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:42.793320 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 23:01:42.793452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:01:42.793919 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:42.782934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46683 (local address 127.21.66.65:44937)
0620 23:01:42.783105 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:42.783151 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:42.783676 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:42.784026 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:42.784030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:42.784754 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:42.784993 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:42.784999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.785829 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.785830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:42.786799 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:42.786801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.787250 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.787251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:42.787470 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:42.787987 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:42.788000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:42.789749 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:42.791680 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:42.791688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:42.791690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:42.792110 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:42.792433 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:42.792434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:42.792435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:42.792609 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:42.792914 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:42.792922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:42.793120 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 23:01:42.793280 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 23:01:42.793565 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260620 23:01:42.794744 24402 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46683
I20260620 23:01:42.794880 24402 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:42.795133 24402 heartbeater.cc:511] Master 127.21.66.126:46683 requested a full tablet report, sending...
I20260620 23:01:42.795765 24211 ts_manager.cc:194] Registered new tserver with Master: 7da70e11241c4ef6b1dc3255b67b9807 (127.21.66.65:35089)
I20260620 23:01:42.796458 24211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:44937
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:42.821666 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46683
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:42.909528 24407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:42.909799 24407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:42.909845 24407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:42.911209 24407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:42.911288 24407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:42.911336 24407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:42.913041 24407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:42.913347 24407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:42.913599 24407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:42.919070 24413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:42.919315 24415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:42.919315 24412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:42.919570 24407 server_base.cc:1034] running on GCE node
I20260620 23:01:42.919827 24407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:42.920146 24407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:42.921296 24407 hybrid_clock.cc:648] HybridClock initialized: now 1781996502921281 us; error 31 us; skew 500 ppm
Jun 20 23:01:42 dist-test-slave-qgtk krb5kdc[24166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996502, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:42.924162 24407 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:42.924667 24407 webserver.cc:460] Webserver started at http://127.21.66.66:43657/ using document root <none> and password file <none>
I20260620 23:01:42.924916 24407 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:42.924961 24407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:42.925105 24407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:42.926303 24407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "da2ea419d76c4d6ea69af63cf1805339"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "010bd37fa840f1873f9e629627cf0924"
server_key_iv: "7d25d21d56b8c9835cdd9a01e2628281"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.926695 24407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "da2ea419d76c4d6ea69af63cf1805339"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "010bd37fa840f1873f9e629627cf0924"
server_key_iv: "7d25d21d56b8c9835cdd9a01e2628281"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.928208 24407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:42.929108 24422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:42.929375 24407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:42.929456 24407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "da2ea419d76c4d6ea69af63cf1805339"
format_stamp: "Formatted at 2026-06-20 23:01:42 on dist-test-slave-qgtk"
server_key: "010bd37fa840f1873f9e629627cf0924"
server_key_iv: "7d25d21d56b8c9835cdd9a01e2628281"
server_key_version: "encryptionkey@0"
I20260620 23:01:42.929557 24407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:42.957974 24407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:42.960374 24407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:42.960553 24407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:42.960866 24407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:42.961210 24407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:42.961243 24407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:42.961293 24407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:42.961320 24407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:42.967121 24407 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:40159
I20260620 23:01:42.967175 24535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:40159 every 8 connection(s)
I20260620 23:01:42.967514 24407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:01:42 dist-test-slave-qgtk krb5kdc[24166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996502, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:42.977362 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:42.968271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:38603 (local address 127.21.66.126:46683)
0620 23:01:42.968342 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:42.968345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:42.968576 (+ 231us) server_negotiation.cc:407] Connection header received
0620 23:01:42.969065 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:42.969066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:42.969077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:42.969141 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:42.970157 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.970709 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.971841 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.972031 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.972089 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:42.974560 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:42.974561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:42.974561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:42.974569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:42.975977 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:42.976410 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:42.976411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:42.976411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:42.976466 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:42.976743 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:42.976744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:42.976744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:42.976856 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:42.976910 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:42.977184 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 23:01:42.977292 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:01:42.977516 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 24407
I20260620 23:01:42.977633 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:42.977764 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:42.968250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46683 (local address 127.21.66.66:38603)
0620 23:01:42.968428 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:42.968474 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:42.968977 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:42.969203 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:42.969206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:42.969803 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:42.970008 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:42.970013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.970811 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.970812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:42.971766 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:42.971769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:42.972144 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:42.972145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:42.972247 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:42.972869 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:42.972877 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:42.974463 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:42.976081 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:42.976088 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:42.976090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:42.976333 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:42.976567 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:42.976568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:42.976568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:42.976680 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:42.976916 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:42.976920 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:42.977067 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:01:42.977159 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 23:01:42.977477 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260620 23:01:42.977849 21769 external_mini_cluster.cc:1428] Setting key 2b21f955826adbad15b448bc0de5230e
I20260620 23:01:42.978374 24536 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46683
I20260620 23:01:42.978502 24536 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:42.978782 24536 heartbeater.cc:511] Master 127.21.66.126:46683 requested a full tablet report, sending...
I20260620 23:01:42.979254 24211 ts_manager.cc:194] Registered new tserver with Master: da2ea419d76c4d6ea69af63cf1805339 (127.21.66.66:40159)
I20260620 23:01:42.979789 24211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:38603
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:43.010203 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46683
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:43.102212 24541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:43.102495 24541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:43.102556 24541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:43.104032 24541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:43.104126 24541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:43.104182 24541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:43.105986 24541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:43.106340 24541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:43.106626 24541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:43.112447 24546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:43.112547 24547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:43.112757 24541 server_base.cc:1034] running on GCE node
W20260620 23:01:43.112864 24549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:43.113116 24541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:43.113427 24541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:43.114724 24541 hybrid_clock.cc:648] HybridClock initialized: now 1781996503114697 us; error 33 us; skew 500 ppm
Jun 20 23:01:43 dist-test-slave-qgtk krb5kdc[24166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996503, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:43.117782 24541 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:43.118431 24541 webserver.cc:460] Webserver started at http://127.21.66.67:34257/ using document root <none> and password file <none>
I20260620 23:01:43.118707 24541 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:43.118790 24541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:43.118978 24541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:43.120471 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "5762bd5991284691a7a78cc7ff2c93e6"
format_stamp: "Formatted at 2026-06-20 23:01:43 on dist-test-slave-qgtk"
server_key: "f94c1683fbf1524d3159c82cb5dc37a2"
server_key_iv: "685b92a28d36c4e13d340ee1407202d7"
server_key_version: "encryptionkey@0"
I20260620 23:01:43.120981 24541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "5762bd5991284691a7a78cc7ff2c93e6"
format_stamp: "Formatted at 2026-06-20 23:01:43 on dist-test-slave-qgtk"
server_key: "f94c1683fbf1524d3159c82cb5dc37a2"
server_key_iv: "685b92a28d36c4e13d340ee1407202d7"
server_key_version: "encryptionkey@0"
I20260620 23:01:43.122717 24541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260620 23:01:43.123714 24556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:43.123943 24541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:43.124051 24541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "5762bd5991284691a7a78cc7ff2c93e6"
format_stamp: "Formatted at 2026-06-20 23:01:43 on dist-test-slave-qgtk"
server_key: "f94c1683fbf1524d3159c82cb5dc37a2"
server_key_iv: "685b92a28d36c4e13d340ee1407202d7"
server_key_version: "encryptionkey@0"
I20260620 23:01:43.124118 24541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:43.136247 24541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:43.138891 24541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:43.139017 24541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:43.139228 24541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:43.139552 24541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:43.139590 24541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:43.139612 24541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:43.139658 24541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:43.145567 24541 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:37667
I20260620 23:01:43.145630 24669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:37667 every 8 connection(s)
I20260620 23:01:43.145965 24541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:01:43 dist-test-slave-qgtk krb5kdc[24166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996503, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:43.155905 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 24541
I20260620 23:01:43.156015 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:43.156251 21769 external_mini_cluster.cc:1428] Setting key d3663ca9d1db78671b73e2069ff61d88
I20260620 23:01:43.157693 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:43.146847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:37879 (local address 127.21.66.126:46683)
0620 23:01:43.146939 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:43.146942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:43.147178 (+ 236us) server_negotiation.cc:407] Connection header received
0620 23:01:43.147767 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:43.147769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:43.147787 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:43.147869 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:43.149057 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:43.149630 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:43.151027 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:43.151233 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:43.151294 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:43.154281 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:43.154282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:43.154283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:43.154292 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:43.155790 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:43.156451 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:43.156453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:43.156454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:43.156537 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:43.156874 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:43.156875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:43.156875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:43.157039 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:43.157110 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:43.157493 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 23:01:43.157577 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:01:43.157873 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:43.146809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46683 (local address 127.21.66.67:37879)
0620 23:01:43.147035 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:43.147080 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:43.147625 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:43.147930 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:43.147935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:43.148714 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:43.148937 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:43.148943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:43.149755 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:43.149756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:43.150915 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:43.150919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:43.151323 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:43.151325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:43.151533 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:43.152279 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:43.152289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:43.154096 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:43.155900 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:43.155910 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:43.155913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:43.156292 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:43.156619 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:43.156620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:43.156621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:43.156802 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:43.157108 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:43.157114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:43.157323 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 23:01:43.157472 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 23:01:43.157633 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
I20260620 23:01:43.158387 24670 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46683
I20260620 23:01:43.158522 24670 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:43.158736 24670 heartbeater.cc:511] Master 127.21.66.126:46683 requested a full tablet report, sending...
I20260620 23:01:43.159262 24211 ts_manager.cc:194] Registered new tserver with Master: 5762bd5991284691a7a78cc7ff2c93e6 (127.21.66.67:37667)
I20260620 23:01:43.159641 24211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:37879
I20260620 23:01:43.168458 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:01:43.181509 24250 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:43.180421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.21.66.126:46683)
0620 23:01:43.180503 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:43.180506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:43.180645 (+ 139us) server_negotiation.cc:407] Connection header received
0620 23:01:43.180705 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:43.180706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:43.180716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:43.180766 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:43.181425 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55092: BlockingRecv error: recv got EOF from 127.0.0.1:55092 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:01:43.183079 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 24273
I20260620 23:01:43.187795 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 24407
I20260620 23:01:43.192626 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 24541
I20260620 23:01:43.197306 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 24179
2026-06-20T23:01:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2031 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:43 dist-test-slave-qgtk krb5kdc[24682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:43 dist-test-slave-qgtk krb5kdc[24682](info): set up 2 sockets
Jun 20 23:01:43 dist-test-slave-qgtk krb5kdc[24682](info): commencing operation
krb5kdc: starting...
W20260620 23:01:44.310624 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:44 dist-test-slave-qgtk krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:44.446074 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39827
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:44.530498 24695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:44.530769 24695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:44.530797 24695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:44.532130 24695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:44.532184 24695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:44.532199 24695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:44.532213 24695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:44.532225 24695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:44.533855 24695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:44.534169 24695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:44.534416 24695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:44.539608 24701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:44.539570 24703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:44.539731 24695 server_base.cc:1034] running on GCE node
W20260620 23:01:44.539590 24700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:44.540133 24695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:44.540540 24695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:44.541697 24695 hybrid_clock.cc:648] HybridClock initialized: now 1781996504541661 us; error 50 us; skew 500 ppm
Jun 20 23:01:44 dist-test-slave-qgtk krb5kdc[24682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996504, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:44.549062 24695 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:44.549593 24695 webserver.cc:460] Webserver started at http://127.21.66.126:45295/ using document root <none> and password file <none>
I20260620 23:01:44.549863 24695 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:44.549934 24695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:44.550096 24695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:44.551304 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "cd363fbbdfd548138d07d3141f9cbd5e"
format_stamp: "Formatted at 2026-06-20 23:01:44 on dist-test-slave-qgtk"
server_key: "68d04849762f9399d3ac20c53bfe6d1d"
server_key_iv: "259011877e3813213524d6111b355cb5"
server_key_version: "encryptionkey@0"
I20260620 23:01:44.551901 24695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "cd363fbbdfd548138d07d3141f9cbd5e"
format_stamp: "Formatted at 2026-06-20 23:01:44 on dist-test-slave-qgtk"
server_key: "68d04849762f9399d3ac20c53bfe6d1d"
server_key_iv: "259011877e3813213524d6111b355cb5"
server_key_version: "encryptionkey@0"
I20260620 23:01:44.553772 24695 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 23:01:44.554754 24710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:44.555035 24695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:44.555164 24695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "cd363fbbdfd548138d07d3141f9cbd5e"
format_stamp: "Formatted at 2026-06-20 23:01:44 on dist-test-slave-qgtk"
server_key: "68d04849762f9399d3ac20c53bfe6d1d"
server_key_iv: "259011877e3813213524d6111b355cb5"
server_key_version: "encryptionkey@0"
I20260620 23:01:44.555258 24695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:44.587127 24695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:44.589749 24695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:44.589943 24695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:44.594743 24695 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:39827
I20260620 23:01:44.594843 24762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:39827 every 8 connection(s)
I20260620 23:01:44.595360 24695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:44.596093 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:44.598639 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e: Bootstrap starting.
I20260620 23:01:44.599260 24763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:44.599574 24763 log.cc:824] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:44.600315 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e: No bootstrap required, opened a new log
I20260620 23:01:44.601948 24763 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39827 } }
I20260620 23:01:44.602181 24763 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:44.602240 24763 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd363fbbdfd548138d07d3141f9cbd5e, State: Initialized, Role: FOLLOWER
I20260620 23:01:44.602370 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 24695
I20260620 23:01:44.602461 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:44.602412 24763 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39827 } }
I20260620 23:01:44.602504 24763 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:44.602548 24763 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:44.602600 24763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:44.602654 21769 external_mini_cluster.cc:1428] Setting key 42fa62635c05b9b3f9860aef11d44737
I20260620 23:01:44.603801 24763 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39827 } }
I20260620 23:01:44.603895 24763 leader_election.cc:302] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd363fbbdfd548138d07d3141f9cbd5e; no voters:
I20260620 23:01:44.604058 24763 leader_election.cc:288] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:44.604147 24768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:44.604360 24763 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:44.604380 24768 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [term 1 LEADER]: Becoming Leader. State: Replica: cd363fbbdfd548138d07d3141f9cbd5e, State: Running, Role: LEADER
I20260620 23:01:44.605195 24768 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39827 } }
I20260620 23:01:44.606251 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd363fbbdfd548138d07d3141f9cbd5e. Latest consensus state: current_term: 1 leader_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39827 } } }
I20260620 23:01:44.606352 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:44.606500 24769 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd363fbbdfd548138d07d3141f9cbd5e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39827 } } }
I20260620 23:01:44.606549 24769 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:44.606745 24782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:44.607267 24782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:44.609568 24782 catalog_manager.cc:1347] Generated new cluster ID: 92e5cf47d8794e0891a2d1c0a9e52c18
I20260620 23:01:44.609629 24782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:44 dist-test-slave-qgtk krb5kdc[24682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:44.615144 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:44.603246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.21.66.126:39827)
0620 23:01:44.603509 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:44.603517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:44.603574 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:01:44.603740 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:44.603746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:44.603791 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:44.603995 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:44.605083 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:44.606664 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:44.607789 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:44.608460 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:44.608594 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:44.611250 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:44.611254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:44.611255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:44.611268 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:44.613115 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:44.613561 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:44.613564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:44.613565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:44.613687 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:44.613921 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:44.613922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:44.613922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:44.614269 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:44.614457 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:44.614584 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 23:01:44.614742 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
I20260620 23:01:44.616837 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:44.617674 24782 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:44.618450 24782 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:44.624138 24782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd363fbbdfd548138d07d3141f9cbd5e: Generated new TSK 0
I20260620 23:01:44.624370 24782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:44.633301 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:44.628686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.21.66.126:39827)
0620 23:01:44.628758 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:44.628760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:44.628865 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:01:44.629101 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:44.629102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:44.629115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:44.629192 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:44.629834 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:44.630558 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:44.631406 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:44.631660 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:44.631745 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:44.632680 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:44.632682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:44.632682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:44.632894 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:44.632912 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:44.632919 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:44.632986 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:44.633151 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 23:01:44.633186 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:01:44.636235 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 24695
2026-06-20T23:01:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1435 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:44 dist-test-slave-qgtk krb5kdc[24796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:44 dist-test-slave-qgtk krb5kdc[24796](info): set up 2 sockets
Jun 20 23:01:44 dist-test-slave-qgtk krb5kdc[24796](info): commencing operation
krb5kdc: starting...
W20260620 23:01:45.710490 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:45 dist-test-slave-qgtk krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:45.846963 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34797
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:45.933571 24809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:45.933868 24809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:45.933911 24809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:45.935317 24809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:45.935420 24809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:45.935446 24809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:45.935472 24809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:45.935511 24809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:45.937441 24809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:45.937808 24809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:45.938098 24809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:45.943732 24817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:45.943849 24815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:45.943732 24814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:45.944115 24809 server_base.cc:1034] running on GCE node
I20260620 23:01:45.944360 24809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:45.944700 24809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:45.945902 24809 hybrid_clock.cc:648] HybridClock initialized: now 1781996505945897 us; error 46 us; skew 500 ppm
Jun 20 23:01:45 dist-test-slave-qgtk krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996505, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:45.948995 24809 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:45.949585 24809 webserver.cc:460] Webserver started at http://127.21.66.126:38537/ using document root <none> and password file <none>
I20260620 23:01:45.949826 24809 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:45.949874 24809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:45.949983 24809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:45.951326 24809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "1f8b96b223854925975b3793b476c94f"
format_stamp: "Formatted at 2026-06-20 23:01:45 on dist-test-slave-qgtk"
server_key: "d4854b61ba3bd82d1a463b29ecefe1a4"
server_key_iv: "1d3db71943ef6c318c4b4e060ae46d1c"
server_key_version: "encryptionkey@0"
I20260620 23:01:45.951694 24809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "1f8b96b223854925975b3793b476c94f"
format_stamp: "Formatted at 2026-06-20 23:01:45 on dist-test-slave-qgtk"
server_key: "d4854b61ba3bd82d1a463b29ecefe1a4"
server_key_iv: "1d3db71943ef6c318c4b4e060ae46d1c"
server_key_version: "encryptionkey@0"
I20260620 23:01:45.953343 24809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:01:45.954437 24824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:45.954774 24809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:45.954866 24809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "1f8b96b223854925975b3793b476c94f"
format_stamp: "Formatted at 2026-06-20 23:01:45 on dist-test-slave-qgtk"
server_key: "d4854b61ba3bd82d1a463b29ecefe1a4"
server_key_iv: "1d3db71943ef6c318c4b4e060ae46d1c"
server_key_version: "encryptionkey@0"
I20260620 23:01:45.954982 24809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:45.973850 24809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:45.976341 24809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:45.976572 24809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:45.981637 24809 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:34797
I20260620 23:01:45.981729 24876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:34797 every 8 connection(s)
I20260620 23:01:45.982164 24809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:45.982492 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 24809
I20260620 23:01:45.982615 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:45.982801 21769 external_mini_cluster.cc:1428] Setting key feaf614b9011f207306c1103c6c5cb8e
I20260620 23:01:45.983557 24877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:45.986955 24877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f: Bootstrap starting.
I20260620 23:01:45.987684 24877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:45.987994 24877 log.cc:824] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:45.988734 24877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f: No bootstrap required, opened a new log
Jun 20 23:01:45 dist-test-slave-qgtk krb5kdc[24796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:45.990759 24877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8b96b223854925975b3793b476c94f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34797 } }
I20260620 23:01:45.990939 24877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:45.990989 24877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8b96b223854925975b3793b476c94f, State: Initialized, Role: FOLLOWER
I20260620 23:01:45.991214 24877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8b96b223854925975b3793b476c94f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34797 } }
I20260620 23:01:45.991331 24877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:45.991417 24877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:45.991480 24877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:45.992094 24877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8b96b223854925975b3793b476c94f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34797 } }
I20260620 23:01:45.992185 24877 leader_election.cc:302] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f8b96b223854925975b3793b476c94f; no voters:
I20260620 23:01:45.992408 24877 leader_election.cc:288] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:45.992492 24882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:45.992695 24877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:45.992729 24882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [term 1 LEADER]: Becoming Leader. State: Replica: 1f8b96b223854925975b3793b476c94f, State: Running, Role: LEADER
I20260620 23:01:45.992925 24882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8b96b223854925975b3793b476c94f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34797 } }
I20260620 23:01:45.993335 24883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f8b96b223854925975b3793b476c94f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8b96b223854925975b3793b476c94f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34797 } } }
I20260620 23:01:45.993443 24883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:45.993628 24884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f8b96b223854925975b3793b476c94f. Latest consensus state: current_term: 1 leader_uuid: "1f8b96b223854925975b3793b476c94f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f8b96b223854925975b3793b476c94f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34797 } } }
I20260620 23:01:45.993723 24884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:45.994238 24891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:45.995013 24891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:45.995546 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:45.983370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.21.66.126:34797)
0620 23:01:45.983632 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:45.983641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:45.983685 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:01:45.984158 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:45.984165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:45.984211 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:45.984328 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:45.985258 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:45.986344 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:45.987450 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:45.988025 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:45.988202 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:45.990926 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:45.990936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:45.990937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:45.990950 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:45.993472 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:45.993991 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:45.993995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:45.993997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:45.994145 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:45.994391 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:45.994392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:45.994392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:45.994759 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:45.994929 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:45.995089 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:01:45.995231 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260620 23:01:45.996788 24891 catalog_manager.cc:1347] Generated new cluster ID: 0fe243420fbc4522bd65dd41b6be2373
I20260620 23:01:45.996850 24891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:46.025154 24891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:46.026023 24891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:46.037460 24891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f8b96b223854925975b3793b476c94f: Generated new TSK 0
I20260620 23:01:46.037719 24891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:46.075354 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34797
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:46.162098 24903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:46.162367 24903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:46.162395 24903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:46.163722 24903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:46.163776 24903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:46.163805 24903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:46.165403 24903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:46.165679 24903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:46.165946 24903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:46.171700 24909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:46.171746 24903 server_base.cc:1034] running on GCE node
W20260620 23:01:46.171676 24908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:46.171681 24911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:46.172075 24903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:46.172330 24903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:46.173483 24903 hybrid_clock.cc:648] HybridClock initialized: now 1781996506173465 us; error 40 us; skew 500 ppm
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:46.176271 24903 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:46.176769 24903 webserver.cc:460] Webserver started at http://127.21.66.65:33141/ using document root <none> and password file <none>
I20260620 23:01:46.177011 24903 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:46.177059 24903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:46.177223 24903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:46.178412 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "d6add6800b354aa88658e076bc6157f1"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "026faa9d2f42da9fa41255f5ce1f2736"
server_key_iv: "6df3c70d55c547d188ee89697d632e7f"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.178781 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "d6add6800b354aa88658e076bc6157f1"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "026faa9d2f42da9fa41255f5ce1f2736"
server_key_iv: "6df3c70d55c547d188ee89697d632e7f"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.180171 24903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:46.181252 24918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.181555 24903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:46.181663 24903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "d6add6800b354aa88658e076bc6157f1"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "026faa9d2f42da9fa41255f5ce1f2736"
server_key_iv: "6df3c70d55c547d188ee89697d632e7f"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.181753 24903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:46.200664 24903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:46.203361 24903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:46.203547 24903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:46.203799 24903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:46.204149 24903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:46.204212 24903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.204247 24903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:46.204276 24903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.210294 24903 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:36945
I20260620 23:01:46.210351 25031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:36945 every 8 connection(s)
I20260620 23:01:46.210723 24903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:46.211242 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 24903
I20260620 23:01:46.211352 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:46.211570 21769 external_mini_cluster.cc:1428] Setting key 284580b70568f0b58e387fdfe4350d1c
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:01:46.222671 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:46.211669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:43403 (local address 127.21.66.126:34797)
0620 23:01:46.211724 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:46.211727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:46.212003 (+ 276us) server_negotiation.cc:407] Connection header received
0620 23:01:46.212580 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:46.212581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:46.212594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:46.212646 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:46.213743 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.214588 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.215827 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.216123 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.216214 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:46.219080 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:46.219081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:46.219082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:46.219091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:46.220748 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.221232 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.221234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.221234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.221300 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.221667 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.221668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.221669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.221862 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:46.221958 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:46.222461 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 23:01:46.222587 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:46.222960 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:46.211597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34797 (local address 127.21.66.65:43403)
0620 23:01:46.211832 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:46.211870 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:46.212460 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:46.212736 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:46.212740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:46.213420 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:46.213636 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:46.213641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.214737 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.214738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:46.215734 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:46.215736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.216238 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.216239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:46.216371 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:46.216962 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:46.216971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:46.218959 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:46.220870 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:46.220882 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:46.220885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:46.221146 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:46.221372 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:46.221373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:46.221373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:46.221522 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:46.221978 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:46.221988 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:46.222250 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 23:01:46.222401 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 23:01:46.222635 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260620 23:01:46.223650 25032 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34797
I20260620 23:01:46.223780 25032 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:46.224027 25032 heartbeater.cc:511] Master 127.21.66.126:34797 requested a full tablet report, sending...
I20260620 23:01:46.224834 24841 ts_manager.cc:194] Registered new tserver with Master: d6add6800b354aa88658e076bc6157f1 (127.21.66.65:36945)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
I20260620 23:01:46.225498 24841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:43403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:46.246951 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34797
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:46.334731 25037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:46.334976 25037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:46.335011 25037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:46.336344 25037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:46.336405 25037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:46.336436 25037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:46.338045 25037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:46.338379 25037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:46.338660 25037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:46.344498 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:46.344512 25043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:46.344580 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:46.344874 25037 server_base.cc:1034] running on GCE node
I20260620 23:01:46.345129 25037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:46.345415 25037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:46.346670 25037 hybrid_clock.cc:648] HybridClock initialized: now 1781996506346703 us; error 78 us; skew 500 ppm
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:46.349468 25037 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:46.350020 25037 webserver.cc:460] Webserver started at http://127.21.66.66:40643/ using document root <none> and password file <none>
I20260620 23:01:46.350287 25037 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:46.350360 25037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:46.350517 25037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:46.351662 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "288d1b7465c6430584fe36e75d708b00"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "4bcc31e2e0118376c6d907d2ef959d5c"
server_key_iv: "51d398ede5020b98796ed31fbe56a33a"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.352049 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "288d1b7465c6430584fe36e75d708b00"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "4bcc31e2e0118376c6d907d2ef959d5c"
server_key_iv: "51d398ede5020b98796ed31fbe56a33a"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.353451 25037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:46.354421 25052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.354916 25037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:46.355077 25037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "288d1b7465c6430584fe36e75d708b00"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "4bcc31e2e0118376c6d907d2ef959d5c"
server_key_iv: "51d398ede5020b98796ed31fbe56a33a"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.355166 25037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:46.366643 25037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:46.369200 25037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:46.369382 25037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:46.369640 25037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:46.369968 25037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:46.370030 25037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.370077 25037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:46.370106 25037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.376658 25037 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:46509
I20260620 23:01:46.376693 25165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:46509 every 8 connection(s)
I20260620 23:01:46.377075 25037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:01:46.382920 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25037
I20260620 23:01:46.383019 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:46.383239 21769 external_mini_cluster.cc:1428] Setting key 61e61bc8ca3ba95cecf32df8c5bfb776
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:46.387878 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:46.377913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:41585 (local address 127.21.66.126:34797)
0620 23:01:46.377997 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:46.378000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:46.378239 (+ 239us) server_negotiation.cc:407] Connection header received
0620 23:01:46.378759 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:46.378760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:46.378770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:46.378825 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:46.379811 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.380361 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.381424 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.381653 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.381724 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:46.384236 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:46.384238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:46.384238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:46.384247 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:46.386279 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.386780 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.386782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.386782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.386863 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.387169 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.387170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.387171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.387302 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:46.387379 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:46.387676 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 23:01:46.387766 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:01:46.388101 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:46.377867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34797 (local address 127.21.66.66:41585)
0620 23:01:46.378070 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:46.378110 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:46.378658 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:46.378886 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:46.378889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:46.379486 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:46.379691 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:46.379696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.380461 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.380461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:46.381322 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:46.381325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.381755 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.381756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:46.381866 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:46.382455 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:46.382464 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:46.384141 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:46.386372 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:46.386382 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:46.386385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:46.386676 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:46.386950 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:46.386951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:46.386951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:46.387089 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:46.387368 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:46.387376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:46.387536 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 23:01:46.387655 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:01:46.387834 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260620 23:01:46.388567 25166 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34797
I20260620 23:01:46.388667 25166 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:46.388863 25166 heartbeater.cc:511] Master 127.21.66.126:34797 requested a full tablet report, sending...
I20260620 23:01:46.389261 24841 ts_manager.cc:194] Registered new tserver with Master: 288d1b7465c6430584fe36e75d708b00 (127.21.66.66:46509)
I20260620 23:01:46.389658 24841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:41585
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:46.414321 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34797
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:46.500401 25171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:46.500658 25171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:46.500686 25171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:46.501930 25171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:46.501986 25171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:46.502014 25171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:46.503643 25171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:46.503927 25171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:46.504201 25171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:46.509662 25177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:46.509763 25179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:46.509958 25171 server_base.cc:1034] running on GCE node
W20260620 23:01:46.509665 25176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:46.510351 25171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:46.510581 25171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:46.511763 25171 hybrid_clock.cc:648] HybridClock initialized: now 1781996506511734 us; error 41 us; skew 500 ppm
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:46.514472 25171 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:46.514967 25171 webserver.cc:460] Webserver started at http://127.21.66.67:34499/ using document root <none> and password file <none>
I20260620 23:01:46.515208 25171 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:46.515280 25171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:46.515434 25171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:46.516520 25171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "42214a0b5cbd4dd1b4f4872680afb661"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "bc0ea2b6254d265c0ef4634b7be44256"
server_key_iv: "3da917522ec1b2ce42c10fa86fb69631"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.516896 25171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "42214a0b5cbd4dd1b4f4872680afb661"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "bc0ea2b6254d265c0ef4634b7be44256"
server_key_iv: "3da917522ec1b2ce42c10fa86fb69631"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.518487 25171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:01:46.519512 25186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.519757 25171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:46.519884 25171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "42214a0b5cbd4dd1b4f4872680afb661"
format_stamp: "Formatted at 2026-06-20 23:01:46 on dist-test-slave-qgtk"
server_key: "bc0ea2b6254d265c0ef4634b7be44256"
server_key_iv: "3da917522ec1b2ce42c10fa86fb69631"
server_key_version: "encryptionkey@0"
I20260620 23:01:46.519974 25171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:46.531333 25171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:46.533810 25171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:46.534005 25171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:46.534335 25171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:46.534686 25171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:46.534744 25171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.534783 25171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:46.534832 25171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:46.541394 25171 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:41117
I20260620 23:01:46.541432 25299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:41117 every 8 connection(s)
I20260620 23:01:46.541783 25171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:46.549889 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25171
I20260620 23:01:46.550025 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:46.550273 21769 external_mini_cluster.cc:1428] Setting key 9624889c0f670c7624de496151ce687c
I20260620 23:01:46.553041 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:46.542574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:49925 (local address 127.21.66.126:34797)
0620 23:01:46.542694 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:46.542696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:46.542909 (+ 213us) server_negotiation.cc:407] Connection header received
0620 23:01:46.543414 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:46.543416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:46.543427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:46.543493 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:46.544516 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.545125 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.546582 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.546818 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.546892 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:46.549428 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:46.549430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:46.549431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:46.549440 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:46.551573 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.552038 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.552039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.552040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.552100 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.552370 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.552371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.552372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.552499 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:46.552578 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:46.552866 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 23:01:46.552947 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:01:46.553289 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:46.542502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34797 (local address 127.21.66.67:49925)
0620 23:01:46.542756 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:46.542799 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:46.543306 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:46.543578 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:46.543582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:46.544200 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:46.544407 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:46.544412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.545243 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.545244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:46.546396 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:46.546399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.546933 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.546934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:46.547107 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:46.547751 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:46.547759 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:46.549296 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:46.551658 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:46.551667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:46.551669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:46.551946 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:46.552189 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:46.552190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:46.552190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:46.552303 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:46.552567 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:46.552574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:46.552727 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 23:01:46.552835 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:01:46.553005 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":161,"threads_started":1}
I20260620 23:01:46.553722 25300 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34797
I20260620 23:01:46.553817 25300 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:46.553958 25300 heartbeater.cc:511] Master 127.21.66.126:34797 requested a full tablet report, sending...
I20260620 23:01:46.554370 24841 ts_manager.cc:194] Registered new tserver with Master: 42214a0b5cbd4dd1b4f4872680afb661 (127.21.66.67:41117)
I20260620 23:01:46.554720 24841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:49925
I20260620 23:01:46.562875 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[24796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996506, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:46.588083 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:46.579520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.21.66.126:34797)
0620 23:01:46.579681 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:46.579683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:46.579788 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:01:46.579900 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:46.579902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:46.579921 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:46.579994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:46.580722 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.581271 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.582160 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:46.582377 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:46.582475 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:46.585164 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:46.585166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:46.585166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:46.585176 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:46.586873 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.587267 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.587268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.587269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.587333 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:46.587525 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:46.587526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:46.587526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:46.587687 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:46.587800 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:46.587895 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 23:01:46.587977 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 23:01:46.588352 24841 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43598
I20260620 23:01:46.588893 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 24903
I20260620 23:01:46.593466 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25037
I20260620 23:01:46.598186 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25171
I20260620 23:01:46.602566 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 24809
2026-06-20T23:01:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1968 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[25312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[25312](info): set up 2 sockets
Jun 20 23:01:46 dist-test-slave-qgtk krb5kdc[25312](info): commencing operation
krb5kdc: starting...
W20260620 23:01:47.674733 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:47 dist-test-slave-qgtk krb5kdc[25312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:47.805697 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:42899
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:42899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:47.891531 25325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:47.891829 25325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:47.891873 25325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:47.893174 25325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:47.893258 25325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:47.893282 25325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:47.893301 25325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:47.893329 25325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:47.895056 25325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:42899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:47.895387 25325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:47.895653 25325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:01:47.901290 25325 server_base.cc:1034] running on GCE node
W20260620 23:01:47.901252 25333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:47.901237 25331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:47.901263 25330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:47.901772 25325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:47.902240 25325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:47.903424 25325 hybrid_clock.cc:648] HybridClock initialized: now 1781996507903410 us; error 49 us; skew 500 ppm
Jun 20 23:01:47 dist-test-slave-qgtk krb5kdc[25312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996507, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:47.906420 25325 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:47.906963 25325 webserver.cc:460] Webserver started at http://127.21.66.126:45101/ using document root <none> and password file <none>
I20260620 23:01:47.907256 25325 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:47.907334 25325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:47.907506 25325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:47.908737 25325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "d623a349c4ef434ba5e7471abbed352f"
format_stamp: "Formatted at 2026-06-20 23:01:47 on dist-test-slave-qgtk"
server_key: "1fdf3822ed925b55067071da1f2ac46c"
server_key_iv: "9edd18c51afbd19b4d04176032746641"
server_key_version: "encryptionkey@0"
I20260620 23:01:47.909143 25325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "d623a349c4ef434ba5e7471abbed352f"
format_stamp: "Formatted at 2026-06-20 23:01:47 on dist-test-slave-qgtk"
server_key: "1fdf3822ed925b55067071da1f2ac46c"
server_key_iv: "9edd18c51afbd19b4d04176032746641"
server_key_version: "encryptionkey@0"
I20260620 23:01:47.910895 25325 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 23:01:47.911942 25340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:47.912289 25325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:47.912424 25325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "d623a349c4ef434ba5e7471abbed352f"
format_stamp: "Formatted at 2026-06-20 23:01:47 on dist-test-slave-qgtk"
server_key: "1fdf3822ed925b55067071da1f2ac46c"
server_key_iv: "9edd18c51afbd19b4d04176032746641"
server_key_version: "encryptionkey@0"
I20260620 23:01:47.912609 25325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:47.929139 25325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:47.931999 25325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:47.932276 25325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:47.936498 25325 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:42899
I20260620 23:01:47.936544 25392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:42899 every 8 connection(s)
I20260620 23:01:47.937124 25325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:47.937878 25393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:47.940459 25393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f: Bootstrap starting.
I20260620 23:01:47.941135 25393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:47.941479 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25325
I20260620 23:01:47.941483 25393 log.cc:824] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:47.941636 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:47.941826 21769 external_mini_cluster.cc:1428] Setting key 35f51208c7b8717f2c5a5bf03500ee46
I20260620 23:01:47.942366 25393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f: No bootstrap required, opened a new log
I20260620 23:01:47.944769 25393 raft_consensus.cc:348] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d623a349c4ef434ba5e7471abbed352f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42899 } }
I20260620 23:01:47.945009 25393 raft_consensus.cc:374] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:47.945055 25393 raft_consensus.cc:729] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d623a349c4ef434ba5e7471abbed352f, State: Initialized, Role: FOLLOWER
I20260620 23:01:47.945232 25393 consensus_queue.cc:260] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d623a349c4ef434ba5e7471abbed352f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42899 } }
I20260620 23:01:47.945361 25393 raft_consensus.cc:388] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:47.945411 25393 raft_consensus.cc:482] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:47.945463 25393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:47.946333 25393 raft_consensus.cc:504] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d623a349c4ef434ba5e7471abbed352f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42899 } }
I20260620 23:01:47.946447 25393 leader_election.cc:302] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d623a349c4ef434ba5e7471abbed352f; no voters:
I20260620 23:01:47.946717 25393 leader_election.cc:288] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:47.946753 25398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:47.946910 25398 raft_consensus.cc:686] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [term 1 LEADER]: Becoming Leader. State: Replica: d623a349c4ef434ba5e7471abbed352f, State: Running, Role: LEADER
I20260620 23:01:47.947007 25398 consensus_queue.cc:237] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d623a349c4ef434ba5e7471abbed352f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42899 } }
I20260620 23:01:47.947085 25393 sys_catalog.cc:564] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:47.947414 25400 sys_catalog.cc:455] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d623a349c4ef434ba5e7471abbed352f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d623a349c4ef434ba5e7471abbed352f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42899 } } }
I20260620 23:01:47.947515 25400 sys_catalog.cc:458] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:47.947947 25399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d623a349c4ef434ba5e7471abbed352f. Latest consensus state: current_term: 1 leader_uuid: "d623a349c4ef434ba5e7471abbed352f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d623a349c4ef434ba5e7471abbed352f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42899 } } }
I20260620 23:01:47.948029 25399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:47.948235 25405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:47.948799 25405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:47.950822 25405 catalog_manager.cc:1347] Generated new cluster ID: 0530aae7658444f0a474f32448afe610
I20260620 23:01:47.950881 25405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:47 dist-test-slave-qgtk krb5kdc[25312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:47.956262 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:47.942465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.21.66.126:42899)
0620 23:01:47.942685 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:47.942695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:47.942727 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:01:47.944766 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:47.944776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:47.944844 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:47.945032 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:47.945988 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:47.947485 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:47.948753 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:47.949460 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:47.949577 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:47.952541 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:47.952549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:47.952549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:47.952569 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:47.954660 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:47.955058 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:47.955064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:47.955064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:47.955168 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:47.955381 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:47.955382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:47.955383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:47.955648 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:47.955821 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:47.955940 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:01:47.956097 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260620 23:01:47.967140 25405 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:47.967783 25405 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:47.980820 25405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d623a349c4ef434ba5e7471abbed352f: Generated new TSK 0
I20260620 23:01:47.981096 25405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:48.018486 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42899
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:48.136430 25419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:48.136667 25419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:48.136694 25419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:48.137952 25419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:48.137997 25419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:48.138026 25419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:48.139648 25419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:48.139932 25419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:48.140199 25419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:48.145680 25427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:48.145721 25425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:48.145723 25419 server_base.cc:1034] running on GCE node
W20260620 23:01:48.145680 25424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:48.146411 25419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:48.146790 25419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:48.147960 25419 hybrid_clock.cc:648] HybridClock initialized: now 1781996508147907 us; error 77 us; skew 500 ppm
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:48.150722 25419 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:48.151228 25419 webserver.cc:460] Webserver started at http://127.21.66.65:34487/ using document root <none> and password file <none>
I20260620 23:01:48.151484 25419 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:48.151531 25419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:48.151659 25419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:48.152793 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "f04a2f291917403abc639caab34e735a"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "8add5566a68d11eaf8c2ac2bfe7069df"
server_key_iv: "4c8968eb58afa5cbb8de980995104206"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.153162 25419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "f04a2f291917403abc639caab34e735a"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "8add5566a68d11eaf8c2ac2bfe7069df"
server_key_iv: "4c8968eb58afa5cbb8de980995104206"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.154816 25419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:48.155952 25434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.156354 25419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:48.156461 25419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "f04a2f291917403abc639caab34e735a"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "8add5566a68d11eaf8c2ac2bfe7069df"
server_key_iv: "4c8968eb58afa5cbb8de980995104206"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.156575 25419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:48.177151 25419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:48.179850 25419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:48.180006 25419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:48.180302 25419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:48.180640 25419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:48.180696 25419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.180730 25419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:48.180751 25419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.187392 25419 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:35459
I20260620 23:01:48.187449 25547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:35459 every 8 connection(s)
I20260620 23:01:48.187842 25419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:48.195313 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25419
I20260620 23:01:48.195411 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:48.195657 21769 external_mini_cluster.cc:1428] Setting key a0f77f4c8ca73bc0d2e88601d45a43f5
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:48.200721 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.188903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:52223 (local address 127.21.66.126:42899)
0620 23:01:48.188975 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:48.188978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:48.189287 (+ 309us) server_negotiation.cc:407] Connection header received
0620 23:01:48.190531 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:48.190533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:48.190552 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:48.190619 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:48.191448 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.192037 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.193199 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.193436 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.193536 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.196813 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:48.196814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:48.196815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:48.196827 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:48.198806 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.199388 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.199390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.199391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.199464 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.199730 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.199731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.199732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.199909 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:48.199988 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:48.200375 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 23:01:48.200552 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:01:48.200752 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.188823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42899 (local address 127.21.66.65:52223)
0620 23:01:48.189048 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:48.189089 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:48.190393 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:48.190743 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:48.190747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:48.191070 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:48.191323 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:48.191329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.192152 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.192154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:48.193087 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:48.193090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.193550 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.193551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:48.193708 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.194448 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:48.194459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:48.196697 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:48.198926 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:48.198935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:48.198938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:48.199295 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:48.199537 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:48.199538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:48.199538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:48.199653 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:48.200016 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:48.200023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:48.200169 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 23:01:48.200319 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 23:01:48.200534 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260620 23:01:48.201550 25548 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42899
I20260620 23:01:48.201702 25548 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:48.201947 25548 heartbeater.cc:511] Master 127.21.66.126:42899 requested a full tablet report, sending...
I20260620 23:01:48.202721 25357 ts_manager.cc:194] Registered new tserver with Master: f04a2f291917403abc639caab34e735a (127.21.66.65:35459)
I20260620 23:01:48.203409 25357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:52223
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:48.226250 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42899
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:48.312446 25553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:48.312714 25553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:48.312757 25553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:48.313987 25553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:48.314066 25553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:48.314116 25553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:48.315716 25553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:48.316038 25553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:48.316293 25553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:48.321770 25558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:48.321810 25561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:48.321961 25553 server_base.cc:1034] running on GCE node
W20260620 23:01:48.322001 25559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:48.322302 25553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:48.322597 25553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:48.323800 25553 hybrid_clock.cc:648] HybridClock initialized: now 1781996508323794 us; error 80 us; skew 500 ppm
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:48.326895 25553 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:48.327472 25553 webserver.cc:460] Webserver started at http://127.21.66.66:38507/ using document root <none> and password file <none>
I20260620 23:01:48.327724 25553 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:48.327798 25553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:48.327951 25553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:48.329082 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "40202756ead44a7e8e67fd16d29cd66e"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "d77f779e33d5c089fba5833e16b55298"
server_key_iv: "795b11ef563d03d51d1a24706fc37dcd"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.329478 25553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "40202756ead44a7e8e67fd16d29cd66e"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "d77f779e33d5c089fba5833e16b55298"
server_key_iv: "795b11ef563d03d51d1a24706fc37dcd"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.331001 25553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:48.331864 25568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.332149 25553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:48.332263 25553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "40202756ead44a7e8e67fd16d29cd66e"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "d77f779e33d5c089fba5833e16b55298"
server_key_iv: "795b11ef563d03d51d1a24706fc37dcd"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.332345 25553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:48.345916 25553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:48.348349 25553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:48.348555 25553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:48.348837 25553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:48.349169 25553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:48.349202 25553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.349255 25553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:48.349298 25553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.355522 25553 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:38701
I20260620 23:01:48.355604 25681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:38701 every 8 connection(s)
I20260620 23:01:48.355952 25553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:01:48.361682 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25553
I20260620 23:01:48.361819 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:48.361999 21769 external_mini_cluster.cc:1428] Setting key fd555db419ffeaa3d18fa9143c9f78b2
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:48.367621 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.357067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:41143 (local address 127.21.66.126:42899)
0620 23:01:48.357128 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:48.357131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:48.357335 (+ 204us) server_negotiation.cc:407] Connection header received
0620 23:01:48.358452 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:48.358454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:48.358469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:48.358519 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:48.359229 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.359751 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.360865 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.361073 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.361173 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.364215 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:48.364216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:48.364217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:48.364224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:48.365931 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.366520 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.366521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.366522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.366587 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.366885 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.366886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.366886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.367027 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:48.367103 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:48.367418 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:01:48.367518 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:01:48.367846 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.356964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42899 (local address 127.21.66.66:41143)
0620 23:01:48.357152 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:48.357202 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:48.358312 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:48.358595 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:48.358600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:48.358909 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:48.359115 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:48.359120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.359906 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.359907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:48.360766 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:48.360768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.361185 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.361186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:48.361332 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.362063 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:48.362098 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:48.364044 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:48.366059 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:48.366069 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:48.366071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:48.366423 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:48.366678 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:48.366679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:48.366680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:48.366809 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:48.367105 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:48.367112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:48.367275 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:01:48.367382 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:01:48.367569 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260620 23:01:48.368391 25682 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42899
I20260620 23:01:48.368491 25682 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:48.368707 25682 heartbeater.cc:511] Master 127.21.66.126:42899 requested a full tablet report, sending...
I20260620 23:01:48.369180 25357 ts_manager.cc:194] Registered new tserver with Master: 40202756ead44a7e8e67fd16d29cd66e (127.21.66.66:38701)
I20260620 23:01:48.369571 25357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:41143
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:48.393049 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42899
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:48.478761 25687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:48.479004 25687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:48.479048 25687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:48.480285 25687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:48.480357 25687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:48.480404 25687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:48.482352 25687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:48.482642 25687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:48.482913 25687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:48.488400 25692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:48.488400 25695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:48.488420 25687 server_base.cc:1034] running on GCE node
W20260620 23:01:48.488400 25693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:48.488945 25687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:48.489202 25687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:48.490356 25687 hybrid_clock.cc:648] HybridClock initialized: now 1781996508490345 us; error 31 us; skew 500 ppm
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:48.493103 25687 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:48.493593 25687 webserver.cc:460] Webserver started at http://127.21.66.67:45825/ using document root <none> and password file <none>
I20260620 23:01:48.493863 25687 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:48.493911 25687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:48.494072 25687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:48.495241 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "50b6f2be877d47cbb322182e3f6b5f70"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "d1e808b2af63fed377811d6632eb708e"
server_key_iv: "220cedbc17d2139e42dfa97fe3023dff"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.495672 25687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "50b6f2be877d47cbb322182e3f6b5f70"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "d1e808b2af63fed377811d6632eb708e"
server_key_iv: "220cedbc17d2139e42dfa97fe3023dff"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.497133 25687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:48.498328 25702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.498601 25687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:48.498730 25687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "50b6f2be877d47cbb322182e3f6b5f70"
format_stamp: "Formatted at 2026-06-20 23:01:48 on dist-test-slave-qgtk"
server_key: "d1e808b2af63fed377811d6632eb708e"
server_key_iv: "220cedbc17d2139e42dfa97fe3023dff"
server_key_version: "encryptionkey@0"
I20260620 23:01:48.498844 25687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:48.525321 25687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:48.528009 25687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:48.528286 25687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:48.528571 25687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:48.528921 25687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:48.528957 25687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.528981 25687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:48.528993 25687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:48.535127 25687 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:33177
I20260620 23:01:48.535168 25815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:33177 every 8 connection(s)
I20260620 23:01:48.535624 25687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:01:48.538941 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25687
I20260620 23:01:48.539064 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:48.539270 21769 external_mini_cluster.cc:1428] Setting key fbc222988549d4f95dab374c18c15aa4
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:48.547494 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.536589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:42783 (local address 127.21.66.126:42899)
0620 23:01:48.536647 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:48.536649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:48.536961 (+ 312us) server_negotiation.cc:407] Connection header received
0620 23:01:48.537976 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:48.537977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:48.537991 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:48.538039 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:48.538744 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.539334 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.540878 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.541069 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.541129 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.544215 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:48.544216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:48.544217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:48.544226 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:48.545712 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.546254 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.546259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.546259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.546334 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.546773 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.546774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.546775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.546918 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:48.546978 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:48.547350 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 23:01:48.547424 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:01:48.547716 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.536533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42899 (local address 127.21.66.67:42783)
0620 23:01:48.536814 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:48.536855 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:48.537846 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:48.538153 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:48.538158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:48.538435 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:48.538632 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:48.538636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.539442 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.539444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:48.540796 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:48.540799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.541172 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.541173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:48.541301 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.541878 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:48.541887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:48.544124 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:48.545814 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:48.545821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:48.545823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:48.546083 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:48.546481 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:48.546482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:48.546482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:48.546597 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:48.547020 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:48.547026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:48.547171 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:01:48.547325 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 23:01:48.547469 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":160,"threads_started":1}
I20260620 23:01:48.548266 25816 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42899
I20260620 23:01:48.548403 25816 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:48.548651 25816 heartbeater.cc:511] Master 127.21.66.126:42899 requested a full tablet report, sending...
I20260620 23:01:48.549132 25357 ts_manager.cc:194] Registered new tserver with Master: 50b6f2be877d47cbb322182e3f6b5f70 (127.21.66.67:33177)
I20260620 23:01:48.549507 25357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:42783
I20260620 23:01:48.551614 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:48.577097 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.568101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.21.66.65:35459)
0620 23:01:48.568324 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:48.568331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:48.568346 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:01:48.568388 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:48.568391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:48.568409 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:48.568513 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:48.569219 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.569984 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.570827 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.571207 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.571311 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.573942 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:48.573952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:48.573953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:48.573961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:48.575689 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.576078 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.576081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.576081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.576142 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.576411 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.576412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.576412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.576591 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:48.576867 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:48.576923 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:01:48.577001 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:48.586364 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:48.578809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.21.66.126:42899)
0620 23:01:48.578864 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:48.578866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:48.578874 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:48.578916 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:48.578916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:48.578925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:48.578971 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:48.579473 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.579956 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.580711 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:48.580918 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:48.580975 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:48.583606 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:48.583608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:48.583609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:48.583616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:48.585079 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.585446 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.585447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.585448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.585510 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:48.585706 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:48.585707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:48.585708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:48.585872 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:48.586001 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:48.586150 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:01:48.586241 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 23:01:48.586635 25357 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38412
I20260620 23:01:48.587157 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25419
I20260620 23:01:48.592120 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25553
I20260620 23:01:48.596684 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25687
I20260620 23:01:48.600769 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25325
2026-06-20T23:01:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1998 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25828](info): set up 2 sockets
Jun 20 23:01:48 dist-test-slave-qgtk krb5kdc[25828](info): commencing operation
krb5kdc: starting...
W20260620 23:01:49.646466 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:49 dist-test-slave-qgtk krb5kdc[25828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:49.777796 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46337
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:49.859759 25841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:49.859977 25841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:49.860001 25841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:49.861229 25841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:49.861274 25841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:49.861289 25841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:49.861301 25841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:49.861313 25841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:49.862998 25841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:46337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:49.863276 25841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:49.863520 25841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:49.868824 25846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:49.868922 25849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:49.868896 25841 server_base.cc:1034] running on GCE node
W20260620 23:01:49.868824 25847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:49.869351 25841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:49.869767 25841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:49.870988 25841 hybrid_clock.cc:648] HybridClock initialized: now 1781996509870966 us; error 39 us; skew 500 ppm
Jun 20 23:01:49 dist-test-slave-qgtk krb5kdc[25828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996509, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:49.879848 25841 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:49.880272 25841 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:49.880362 25841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:49.880519 25841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:49.881676 25841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "642fbeeada1f4c78add97eaee80e1ff2"
format_stamp: "Formatted at 2026-06-20 23:01:49 on dist-test-slave-qgtk"
server_key: "72d39efe4c80a7baa16817dc4399602c"
server_key_iv: "3cad540c1de3e25a5477e6a065f1f2ad"
server_key_version: "encryptionkey@0"
I20260620 23:01:49.882063 25841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "642fbeeada1f4c78add97eaee80e1ff2"
format_stamp: "Formatted at 2026-06-20 23:01:49 on dist-test-slave-qgtk"
server_key: "72d39efe4c80a7baa16817dc4399602c"
server_key_iv: "3cad540c1de3e25a5477e6a065f1f2ad"
server_key_version: "encryptionkey@0"
I20260620 23:01:49.883723 25841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:49.884727 25855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:49.884944 25841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:49.885070 25841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "642fbeeada1f4c78add97eaee80e1ff2"
format_stamp: "Formatted at 2026-06-20 23:01:49 on dist-test-slave-qgtk"
server_key: "72d39efe4c80a7baa16817dc4399602c"
server_key_iv: "3cad540c1de3e25a5477e6a065f1f2ad"
server_key_version: "encryptionkey@0"
I20260620 23:01:49.885165 25841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:49.908528 25841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:49.911033 25841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:49.911213 25841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:49.915423 25841 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:46337
I20260620 23:01:49.915478 25907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:46337 every 8 connection(s)
I20260620 23:01:49.915819 25841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:49.916829 25908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:49.919296 25908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2: Bootstrap starting.
I20260620 23:01:49.919950 25908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:49.920271 25908 log.cc:824] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:49.921000 25908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2: No bootstrap required, opened a new log
I20260620 23:01:49.923271 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25841
I20260620 23:01:49.923389 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:49.923612 21769 external_mini_cluster.cc:1428] Setting key 58f9b4d466aa8d908b423df669b34a06
I20260620 23:01:49.923528 25908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642fbeeada1f4c78add97eaee80e1ff2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46337 } }
I20260620 23:01:49.923717 25908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:49.923765 25908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642fbeeada1f4c78add97eaee80e1ff2, State: Initialized, Role: FOLLOWER
I20260620 23:01:49.923933 25908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642fbeeada1f4c78add97eaee80e1ff2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46337 } }
I20260620 23:01:49.924027 25908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:49.924067 25908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:49.924292 25908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:49.925599 25908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642fbeeada1f4c78add97eaee80e1ff2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46337 } }
I20260620 23:01:49.925729 25908 leader_election.cc:302] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 642fbeeada1f4c78add97eaee80e1ff2; no voters:
I20260620 23:01:49.925904 25908 leader_election.cc:288] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:49.925932 25913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:49.926105 25913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [term 1 LEADER]: Becoming Leader. State: Replica: 642fbeeada1f4c78add97eaee80e1ff2, State: Running, Role: LEADER
I20260620 23:01:49.926203 25908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:49.926256 25913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642fbeeada1f4c78add97eaee80e1ff2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46337 } }
I20260620 23:01:49.926674 25914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "642fbeeada1f4c78add97eaee80e1ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642fbeeada1f4c78add97eaee80e1ff2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46337 } } }
I20260620 23:01:49.926656 25913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 642fbeeada1f4c78add97eaee80e1ff2. Latest consensus state: current_term: 1 leader_uuid: "642fbeeada1f4c78add97eaee80e1ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642fbeeada1f4c78add97eaee80e1ff2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46337 } } }
I20260620 23:01:49.926828 25913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:49.926815 25914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:49.927105 25922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:49.927949 25922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:49.929618 25922 catalog_manager.cc:1347] Generated new cluster ID: fe57b3560fe14adca9c6d72ee713a712
I20260620 23:01:49.929670 25922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:49 dist-test-slave-qgtk krb5kdc[25828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:49.936419 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:49.924387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.21.66.126:46337)
0620 23:01:49.924649 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:49.924657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:49.924695 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:01:49.924802 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:49.924808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:49.924861 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:49.924977 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:49.925839 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:49.927388 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:49.928616 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:49.929214 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:49.929332 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:49.932171 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:49.932175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:49.932176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:49.932189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:49.934389 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:49.934837 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:49.934841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:49.934842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:49.934974 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:49.935198 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:49.935199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:49.935199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:49.935611 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:49.935760 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:49.936033 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:01:49.936192 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260620 23:01:49.951436 25922 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:49.952255 25922 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:49.965031 25922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2: Generated new TSK 0
I20260620 23:01:49.965292 25922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:49.999468 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46337
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:50.089262 25934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:50.089488 25934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:50.089514 25934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:50.090811 25934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:50.090859 25934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:50.090889 25934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:50.092393 25934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:50.092594 25934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:50.092782 25934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:01:50.098490 25934 server_base.cc:1034] running on GCE node
W20260620 23:01:50.098417 25940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:50.098417 25939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:50.098421 25942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:50.099021 25934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:50.099381 25934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:50.100586 25934 hybrid_clock.cc:648] HybridClock initialized: now 1781996510100575 us; error 31 us; skew 500 ppm
Jun 20 23:01:50 dist-test-slave-qgtk krb5kdc[25828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996510, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:50.103820 25934 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:50.104216 25934 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:50.104310 25934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:50.104480 25934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:50.105762 25934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "2fd0ac924bfb4914ac1e648b09a90b72"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "54ca85578970923b2cacf634f87973ea"
server_key_iv: "44cead275fde6b93ff8d5140458f20ff"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.106217 25934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "2fd0ac924bfb4914ac1e648b09a90b72"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "54ca85578970923b2cacf634f87973ea"
server_key_iv: "44cead275fde6b93ff8d5140458f20ff"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.107600 25934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:50.108567 25948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.108867 25934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:50.108985 25934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "2fd0ac924bfb4914ac1e648b09a90b72"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "54ca85578970923b2cacf634f87973ea"
server_key_iv: "44cead275fde6b93ff8d5140458f20ff"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.109072 25934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:50.127389 25934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:50.129769 25934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:50.129952 25934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:50.130227 25934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:50.130565 25934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:50.130621 25934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.130760 25934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:50.130796 25934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.136554 25934 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:42307
I20260620 23:01:50.136643 26061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:42307 every 8 connection(s)
I20260620 23:01:50.136930 25934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:01:50 dist-test-slave-qgtk krb5kdc[25828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996510, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:50.145843 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 25934
I20260620 23:01:50.145998 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:50.146255 21769 external_mini_cluster.cc:1428] Setting key 7ee0af7da35ab8110686dc1ed25359c0
I20260620 23:01:50.148157 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.137782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:43589 (local address 127.21.66.126:46337)
0620 23:01:50.137825 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.137827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.138015 (+ 188us) server_negotiation.cc:407] Connection header received
0620 23:01:50.138557 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.138558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.138567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.138627 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:50.139582 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.140142 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.141259 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.141498 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.141566 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.144472 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:50.144473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:50.144474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:50.144483 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:50.146448 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.146981 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.146983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.146983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.147066 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.147367 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.147368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.147369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.147502 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:50.147676 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.147931 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.148039 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:01:50.148488 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.137697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46337 (local address 127.21.66.65:43589)
0620 23:01:50.137877 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.137921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.138471 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.138687 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.138690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.138959 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:50.139459 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.139464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.140241 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.140242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.141165 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.141167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.141739 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.141741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.141920 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.142700 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:50.142713 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:50.144367 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:50.146544 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:50.146554 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:50.146557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:50.146892 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:50.147151 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:50.147152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:50.147153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:50.147285 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:50.147575 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:50.147583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:50.147779 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.147909 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.148176 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260620 23:01:50.149024 26062 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46337
I20260620 23:01:50.149138 26062 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:50.149315 26062 heartbeater.cc:511] Master 127.21.66.126:46337 requested a full tablet report, sending...
I20260620 23:01:50.149860 25872 ts_manager.cc:194] Registered new tserver with Master: 2fd0ac924bfb4914ac1e648b09a90b72 (127.21.66.65:42307)
I20260620 23:01:50.150625 25872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:43589
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:50.174000 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46337
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:50.259559 26067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:50.259792 26067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:50.259819 26067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:50.261049 26067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:50.261102 26067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:50.261133 26067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:50.262805 26067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:50.263072 26067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:50.263314 26067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:50.268918 26075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:50.268944 26072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:50.268918 26073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:50.269337 26067 server_base.cc:1034] running on GCE node
I20260620 23:01:50.269600 26067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:50.269884 26067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:50.271075 26067 hybrid_clock.cc:648] HybridClock initialized: now 1781996510271039 us; error 56 us; skew 500 ppm
Jun 20 23:01:50 dist-test-slave-qgtk krb5kdc[25828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996510, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:50.273953 26067 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:50.274369 26067 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:50.274425 26067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:50.274521 26067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:50.275599 26067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "ac22dea4e76648399eb08e1aa7a601af"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "f0bf47b09ff60b8e9c031261d53ea055"
server_key_iv: "f740c2b5f5730adfc21fe1b87c621d07"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.275923 26067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "ac22dea4e76648399eb08e1aa7a601af"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "f0bf47b09ff60b8e9c031261d53ea055"
server_key_iv: "f740c2b5f5730adfc21fe1b87c621d07"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.277308 26067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:50.278187 26081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.278410 26067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:50.278512 26067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "ac22dea4e76648399eb08e1aa7a601af"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "f0bf47b09ff60b8e9c031261d53ea055"
server_key_iv: "f740c2b5f5730adfc21fe1b87c621d07"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.278581 26067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:50.288085 26067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:50.290763 26067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:50.290892 26067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:50.291090 26067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:50.291457 26067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:50.291500 26067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.291524 26067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:50.291571 26067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.297394 26067 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:33081
I20260620 23:01:50.297467 26194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:33081 every 8 connection(s)
I20260620 23:01:50.297773 26067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:01:50.299696 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 26067
I20260620 23:01:50.299817 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:50.300045 21769 external_mini_cluster.cc:1428] Setting key da956d9ab5dc21a4b629384bff148a7f
Jun 20 23:01:50 dist-test-slave-qgtk krb5kdc[25828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996510, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
I20260620 23:01:50.309537 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.298857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:59589 (local address 127.21.66.126:46337)
0620 23:01:50.298963 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.298965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.299216 (+ 251us) server_negotiation.cc:407] Connection header received
0620 23:01:50.300002 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.300003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.300017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.300079 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:50.301346 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.301930 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.303089 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.303314 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.303407 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.305993 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:50.305995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:50.305995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:50.306013 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:50.307769 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.308336 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.308338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.308338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.308416 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.308758 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.308759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.308759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.308883 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:50.308942 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.309320 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.309437 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:01:50.309680 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.298856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46337 (local address 127.21.66.66:59589)
0620 23:01:50.299084 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.299124 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.299819 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.300187 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.300192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.300502 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:50.301219 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.301227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.302056 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.302057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.302978 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.302980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.303417 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.303419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.303551 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.304129 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:50.304144 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:50.305863 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:50.307870 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:50.307880 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:50.307884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:50.308203 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:50.308497 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:50.308498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:50.308499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:50.308666 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:50.308998 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:50.309005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:50.309150 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.309255 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.309466 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":110,"threads_started":1}
I20260620 23:01:50.310356 26195 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46337
I20260620 23:01:50.310448 26195 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:50.310582 26195 heartbeater.cc:511] Master 127.21.66.126:46337 requested a full tablet report, sending...
I20260620 23:01:50.311033 25872 ts_manager.cc:194] Registered new tserver with Master: ac22dea4e76648399eb08e1aa7a601af (127.21.66.66:33081)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
I20260620 23:01:50.311448 25872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:59589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:50.330727 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46337
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:34459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:50.419445 26200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:50.419697 26200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:50.419728 26200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:50.421538 26200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:50.421622 26200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:50.421682 26200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:50.423682 26200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:34459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:50.423925 26200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:50.424201 26200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:50.430696 26205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:50.430780 26200 server_base.cc:1034] running on GCE node
W20260620 23:01:50.430927 26208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:50.430783 26206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:50.431311 26200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:50.431602 26200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:50.432821 26200 hybrid_clock.cc:648] HybridClock initialized: now 1781996510432805 us; error 32 us; skew 500 ppm
Jun 20 23:01:50 dist-test-slave-qgtk krb5kdc[25828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996510, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:50.435617 26200 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:50.436039 26200 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:50.436094 26200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:50.436192 26200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:50.437292 26200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "e21fbac586e1425482c566ac6c3a6d96"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "27ea41d472e2d57a2dfabe38905e943f"
server_key_iv: "7ab3a9d0b5727d6c985e7b0432baf033"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.437597 26200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "e21fbac586e1425482c566ac6c3a6d96"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "27ea41d472e2d57a2dfabe38905e943f"
server_key_iv: "7ab3a9d0b5727d6c985e7b0432baf033"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.439077 26200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:50.439911 26214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.440146 26200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:50.440232 26200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "e21fbac586e1425482c566ac6c3a6d96"
format_stamp: "Formatted at 2026-06-20 23:01:50 on dist-test-slave-qgtk"
server_key: "27ea41d472e2d57a2dfabe38905e943f"
server_key_iv: "7ab3a9d0b5727d6c985e7b0432baf033"
server_key_version: "encryptionkey@0"
I20260620 23:01:50.440302 26200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:50.458029 26200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:50.460644 26200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:50.460832 26200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:50.461053 26200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:50.461431 26200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:50.461498 26200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.461545 26200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:50.461571 26200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:50.468618 26200 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:41233
I20260620 23:01:50.468776 26327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:41233 every 8 connection(s)
I20260620 23:01:50.469028 26200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:01:50 dist-test-slave-qgtk krb5kdc[25828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996510, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:50.476365 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 26200
I20260620 23:01:50.476588 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:50.476873 21769 external_mini_cluster.cc:1428] Setting key 0dc06bfe58c8ff5007d09412ba74be15
I20260620 23:01:50.479629 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.469907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:60763 (local address 127.21.66.126:46337)
0620 23:01:50.469979 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.469981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.470246 (+ 265us) server_negotiation.cc:407] Connection header received
0620 23:01:50.470779 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.470780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.470796 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.470856 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:50.471902 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.472471 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.473622 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.473841 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.473907 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.476551 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:50.476552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:50.476552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:50.476563 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:50.478105 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.478597 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.478598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.478599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.478666 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.478951 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.478952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.478952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.479077 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:50.479149 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.479449 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.479532 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:01:50.479853 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.469828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46337 (local address 127.21.66.67:60763)
0620 23:01:50.470053 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.470093 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.470675 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.470920 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.470923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.471218 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:50.471784 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.471791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.472564 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.472565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.473520 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.473523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.473943 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.473944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.474058 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.474714 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:50.474725 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:50.476348 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:50.478248 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:50.478256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:50.478259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:50.478506 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:50.478755 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:50.478756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:50.478756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:50.478880 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:50.479152 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:50.479159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:50.479315 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.479426 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.479596 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260620 23:01:50.480481 26328 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46337
I20260620 23:01:50.480619 26328 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:50.480813 26328 heartbeater.cc:511] Master 127.21.66.126:46337 requested a full tablet report, sending...
I20260620 23:01:50.481302 25872 ts_manager.cc:194] Registered new tserver with Master: e21fbac586e1425482c566ac6c3a6d96 (127.21.66.67:41233)
I20260620 23:01:50.481778 25872 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:60763
I20260620 23:01:50.488870 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:01:50.497177 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.490359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.21.66.126:46337)
0620 23:01:50.490435 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.490438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.490546 (+ 108us) server_negotiation.cc:407] Connection header received
0620 23:01:50.490595 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.490596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.490607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.490656 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:50.491239 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.491758 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.492553 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.492733 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.492789 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.494073 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:50.494075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:50.494075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:50.494082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:50.495610 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.496060 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.496061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.496061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.496112 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:50.496356 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:50.496358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:50.496358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:50.496533 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:50.496627 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.496928 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.497007 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:01:50.499181 25872 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:50.500804 25872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:50.507907 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.503457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:33081 (local address 127.0.0.1:53270)
0620 23:01:50.503865 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.503873 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.503908 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.504094 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.504097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.504101 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:50.504209 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.504211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.505636 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.505637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.506599 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.506601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.507618 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.507620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.507698 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.507704 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.507766 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.507812 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":31,"thread_start_us":90,"threads_started":1}
I20260620 23:01:50.507921 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.503697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.21.66.67:41233)
0620 23:01:50.504245 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.504250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.504260 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:50.504297 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.504299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.504316 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.504388 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:50.504601 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.505304 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.506874 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.507568 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.507629 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.507679 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.507793 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.507837 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":83,"threads_started":1}
I20260620 23:01:50.507997 26342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.503566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.21.66.65:42307)
0620 23:01:50.503903 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.503909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.503920 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:50.503963 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.503966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.503982 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.504072 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:50.504317 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.505490 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.506901 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.507410 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.507501 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.507579 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.507860 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.507915 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":81,"threads_started":1}
I20260620 23:01:50.508040 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.503454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42307 (local address 127.0.0.1:55750)
0620 23:01:50.503623 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.503667 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.503737 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.504094 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.504097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.504100 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:50.504222 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.504224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.505595 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.505596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.506778 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.506781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.507512 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.507513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.507601 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.507616 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.507685 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.507748 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":35,"thread_start_us":52,"threads_started":1}
I20260620 23:01:50.508235 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.503457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41233 (local address 127.0.0.1:43048)
0620 23:01:50.503732 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.503742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.503799 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.504472 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.504472 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.504474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:50.504518 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.504520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.505411 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.505413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.506780 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.506781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.507640 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.507641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.507702 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.507705 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.507763 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.507796 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260620 23:01:50.508752 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.503709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.21.66.66:33081)
0620 23:01:50.503903 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.503909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.503947 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:01:50.503981 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.503983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.503996 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.504072 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:50.504306 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.505523 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.506685 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.507447 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.507522 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.507581 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.507876 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.507917 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260620 23:01:50.508816 25996 tablet_service.cc:1467] Processing CreateTablet for tablet 82841f069c8248238dd1befbe9fa89f2 (DEFAULT_TABLE table=test-table [id=34871513de1d47ce8d0c2a263eec2b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:50.508980 26262 tablet_service.cc:1467] Processing CreateTablet for tablet 82841f069c8248238dd1befbe9fa89f2 (DEFAULT_TABLE table=test-table [id=34871513de1d47ce8d0c2a263eec2b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:50.509204 25996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82841f069c8248238dd1befbe9fa89f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:50.509423 26262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82841f069c8248238dd1befbe9fa89f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:50.509408 26129 tablet_service.cc:1467] Processing CreateTablet for tablet 82841f069c8248238dd1befbe9fa89f2 (DEFAULT_TABLE table=test-table [id=34871513de1d47ce8d0c2a263eec2b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:50.509704 26129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82841f069c8248238dd1befbe9fa89f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:50.512347 26345 tablet_bootstrap.cc:492] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Bootstrap starting.
I20260620 23:01:50.512804 26346 tablet_bootstrap.cc:492] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72: Bootstrap starting.
I20260620 23:01:50.513384 26345 tablet_bootstrap.cc:654] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:50.513428 26346 tablet_bootstrap.cc:654] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:50.514062 26346 log.cc:824] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:50.514071 26345 log.cc:824] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:50.514449 26347 tablet_bootstrap.cc:492] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Bootstrap starting.
I20260620 23:01:50.514909 26346 tablet_bootstrap.cc:492] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72: No bootstrap required, opened a new log
I20260620 23:01:50.514986 26346 ts_tablet_manager.cc:1397] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:50.515065 26347 tablet_bootstrap.cc:654] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:50.515334 26345 tablet_bootstrap.cc:492] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: No bootstrap required, opened a new log
I20260620 23:01:50.515420 26345 ts_tablet_manager.cc:1397] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:01:50.515669 26347 log.cc:824] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:50.516464 26347 tablet_bootstrap.cc:492] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: No bootstrap required, opened a new log
I20260620 23:01:50.516533 26347 ts_tablet_manager.cc:1397] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:50.516924 26346 raft_consensus.cc:348] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.517093 26346 raft_consensus.cc:374] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:50.517138 26346 raft_consensus.cc:729] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fd0ac924bfb4914ac1e648b09a90b72, State: Initialized, Role: FOLLOWER
I20260620 23:01:50.517273 26346 consensus_queue.cc:260] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.517556 26346 ts_tablet_manager.cc:1428] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:50.517488 26345 raft_consensus.cc:348] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.517704 26062 heartbeater.cc:503] Master 127.21.66.126:46337 was elected leader, sending a full tablet report...
I20260620 23:01:50.517635 26345 raft_consensus.cc:374] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:50.518014 26345 raft_consensus.cc:729] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21fbac586e1425482c566ac6c3a6d96, State: Initialized, Role: FOLLOWER
I20260620 23:01:50.518186 26345 consensus_queue.cc:260] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.518493 26345 ts_tablet_manager.cc:1428] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:50.518502 26328 heartbeater.cc:503] Master 127.21.66.126:46337 was elected leader, sending a full tablet report...
I20260620 23:01:50.518652 26347 raft_consensus.cc:348] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.518843 26347 raft_consensus.cc:374] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:50.518891 26347 raft_consensus.cc:729] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac22dea4e76648399eb08e1aa7a601af, State: Initialized, Role: FOLLOWER
I20260620 23:01:50.519004 26347 consensus_queue.cc:260] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.519330 26195 heartbeater.cc:503] Master 127.21.66.126:46337 was elected leader, sending a full tablet report...
I20260620 23:01:50.519564 26347 ts_tablet_manager.cc:1428] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 23:01:50.548512 26196 tablet.cc:2369] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:01:50.637722 26063 tablet.cc:2369] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:01:50.719668 26329 tablet.cc:2369] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:01:50.801532 26351 raft_consensus.cc:482] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:50.801671 26351 raft_consensus.cc:504] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.802075 26351 leader_election.cc:288] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e21fbac586e1425482c566ac6c3a6d96 (127.21.66.67:41233), ac22dea4e76648399eb08e1aa7a601af (127.21.66.66:33081)
I20260620 23:01:50.805156 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.802333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:33081 (local address 127.21.66.65:39887)
0620 23:01:50.802512 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.802529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.802610 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.802765 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.802766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.802767 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:50.802856 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.802859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.803495 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.803496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.804279 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.804281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.804873 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.804874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.804950 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.804953 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.805016 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.805062 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260620 23:01:50.805138 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.802409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:39887 (local address 127.21.66.66:33081)
0620 23:01:50.802465 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.802468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.802597 (+ 129us) server_negotiation.cc:407] Connection header received
0620 23:01:50.802676 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.802677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.802686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.802760 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:50.802937 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.803409 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.804393 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.804764 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.804867 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.804876 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.805037 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.805072 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:01:50.805325 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.802440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:39011 (local address 127.21.66.67:41233)
0620 23:01:50.802489 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.802491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.802536 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:01:50.802642 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.802644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.802656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.802742 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:50.802923 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.803454 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.804507 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.804929 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.805024 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.805032 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.805210 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.805258 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:01:50.805322 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.802319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:41233 (local address 127.21.66.65:39011)
0620 23:01:50.802467 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.802480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:50.802554 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:50.802746 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:50.802753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:50.802756 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:50.802843 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.802844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.803553 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.803554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.804405 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:50.804407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.805022 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.805023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:50.805117 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.805120 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:50.805190 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:01:50.805232 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260620 23:01:50.805661 26282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82841f069c8248238dd1befbe9fa89f2" candidate_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e21fbac586e1425482c566ac6c3a6d96" is_pre_election: true
I20260620 23:01:50.805658 26149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82841f069c8248238dd1befbe9fa89f2" candidate_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac22dea4e76648399eb08e1aa7a601af" is_pre_election: true
I20260620 23:01:50.805823 26149 raft_consensus.cc:2454] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fd0ac924bfb4914ac1e648b09a90b72 in term 0.
I20260620 23:01:50.805830 26282 raft_consensus.cc:2454] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fd0ac924bfb4914ac1e648b09a90b72 in term 0.
I20260620 23:01:50.806139 25951 leader_election.cc:302] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd0ac924bfb4914ac1e648b09a90b72, e21fbac586e1425482c566ac6c3a6d96; no voters:
I20260620 23:01:50.806313 26351 raft_consensus.cc:2781] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:50.806406 26351 raft_consensus.cc:482] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:50.806450 26351 raft_consensus.cc:3037] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:50.807261 26351 raft_consensus.cc:504] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.807428 26351 leader_election.cc:288] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [CANDIDATE]: Term 1 election: Requested vote from peers e21fbac586e1425482c566ac6c3a6d96 (127.21.66.67:41233), ac22dea4e76648399eb08e1aa7a601af (127.21.66.66:33081)
I20260620 23:01:50.807626 26149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82841f069c8248238dd1befbe9fa89f2" candidate_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac22dea4e76648399eb08e1aa7a601af"
I20260620 23:01:50.807646 26282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82841f069c8248238dd1befbe9fa89f2" candidate_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e21fbac586e1425482c566ac6c3a6d96"
I20260620 23:01:50.807726 26149 raft_consensus.cc:3037] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:50.807724 26282 raft_consensus.cc:3037] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:50.808456 26149 raft_consensus.cc:2454] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fd0ac924bfb4914ac1e648b09a90b72 in term 1.
I20260620 23:01:50.808518 26282 raft_consensus.cc:2454] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fd0ac924bfb4914ac1e648b09a90b72 in term 1.
I20260620 23:01:50.808701 25951 leader_election.cc:302] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fd0ac924bfb4914ac1e648b09a90b72, e21fbac586e1425482c566ac6c3a6d96; no voters:
I20260620 23:01:50.808904 26351 raft_consensus.cc:2781] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:50.809098 26351 raft_consensus.cc:686] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [term 1 LEADER]: Becoming Leader. State: Replica: 2fd0ac924bfb4914ac1e648b09a90b72, State: Running, Role: LEADER
I20260620 23:01:50.809219 26351 consensus_queue.cc:237] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } }
I20260620 23:01:50.810276 25872 catalog_manager.cc:5495] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fd0ac924bfb4914ac1e648b09a90b72 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fd0ac924bfb4914ac1e648b09a90b72" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 } health_report { overall_health: UNKNOWN } } }
I20260620 23:01:50.862533 26342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.859844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.21.66.65:42307)
0620 23:01:50.859916 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:50.859918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:50.859927 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:50.859966 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:50.859967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:50.859993 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:50.860061 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:50.860206 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.860682 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.861507 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:50.861707 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:50.861762 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:50.862101 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:50.862247 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:50.862330 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:50.862420 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 23:01:50.862441 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:01:50.864759 26282 raft_consensus.cc:1261] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 1 FOLLOWER]: Refusing update from remote peer 2fd0ac924bfb4914ac1e648b09a90b72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:50.864872 26149 raft_consensus.cc:1261] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 1 FOLLOWER]: Refusing update from remote peer 2fd0ac924bfb4914ac1e648b09a90b72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:50.865222 26356 consensus_queue.cc:1035] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e21fbac586e1425482c566ac6c3a6d96" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:50.865377 26351 consensus_queue.cc:1035] T 82841f069c8248238dd1befbe9fa89f2 P 2fd0ac924bfb4914ac1e648b09a90b72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac22dea4e76648399eb08e1aa7a601af" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:50.870846 25872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:50.870960 25872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:50.871696 25872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 642fbeeada1f4c78add97eaee80e1ff2: Sending DeleteTablet for 3 replicas of tablet 82841f069c8248238dd1befbe9fa89f2
I20260620 23:01:50.872040 26262 tablet_service.cc:1514] Processing DeleteTablet for tablet 82841f069c8248238dd1befbe9fa89f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:50 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:43048
I20260620 23:01:50.872073 26129 tablet_service.cc:1514] Processing DeleteTablet for tablet 82841f069c8248238dd1befbe9fa89f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:50 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:53270
I20260620 23:01:50.872366 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25934
I20260620 23:01:50.872323 25996 tablet_service.cc:1514] Processing DeleteTablet for tablet 82841f069c8248238dd1befbe9fa89f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:50 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:55750
I20260620 23:01:50.872435 26369 tablet_replica.cc:330] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: stopping tablet replica
I20260620 23:01:50.872555 26369 raft_consensus.cc:2229] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:50.872725 26369 raft_consensus.cc:2258] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:50.873256 26368 tablet_replica.cc:330] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: stopping tablet replica
I20260620 23:01:50.873396 26368 raft_consensus.cc:2229] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:50.873538 26368 raft_consensus.cc:2258] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:50.874075 26369 ts_tablet_manager.cc:1905] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:50.874085 26368 ts_tablet_manager.cc:1905] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:50.876324 26368 ts_tablet_manager.cc:1918] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:50.876324 26369 ts_tablet_manager.cc:1918] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:50.876415 26369 log.cc:1196] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/82841f069c8248238dd1befbe9fa89f2
I20260620 23:01:50.876458 26368 log.cc:1196] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/82841f069c8248238dd1befbe9fa89f2
I20260620 23:01:50.876781 26369 ts_tablet_manager.cc:1939] T 82841f069c8248238dd1befbe9fa89f2 P ac22dea4e76648399eb08e1aa7a601af: Deleting consensus metadata
I20260620 23:01:50.876847 26368 ts_tablet_manager.cc:1939] T 82841f069c8248238dd1befbe9fa89f2 P e21fbac586e1425482c566ac6c3a6d96: Deleting consensus metadata
I20260620 23:01:50.877305 25859 catalog_manager.cc:4841] TS ac22dea4e76648399eb08e1aa7a601af (127.21.66.66:33081): tablet 82841f069c8248238dd1befbe9fa89f2 (table test-table [id=34871513de1d47ce8d0c2a263eec2b07]) successfully deleted
I20260620 23:01:50.877383 25858 catalog_manager.cc:4841] TS e21fbac586e1425482c566ac6c3a6d96 (127.21.66.67:41233): tablet 82841f069c8248238dd1befbe9fa89f2 (table test-table [id=34871513de1d47ce8d0c2a263eec2b07]) successfully deleted
W20260620 23:01:50.878412 25856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:42307 (error 108)
I20260620 23:01:50.878568 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 26067
I20260620 23:01:50.879356 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:50.879206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42307 (local address 127.0.0.1:55764)
0620 23:01:50.879272 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:50.879318 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 23:01:50.879539 25856 catalog_manager.cc:4584] TS 2fd0ac924bfb4914ac1e648b09a90b72 (127.21.66.65:42307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82841f069c8248238dd1befbe9fa89f2: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42307: connect: Connection refused (error 111)
I20260620 23:01:50.883895 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 26200
I20260620 23:01:50.888991 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 25841
2026-06-20T23:01:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2288 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T23:01:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:50Z Disabled control of system clock
I20260620 23:01:50.917629 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34165
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:01:51.001027 26374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:51.001260 26374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:51.001282 26374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:51.002593 26374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:51.002643 26374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:51.002656 26374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:51.002668 26374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:51.002681 26374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:51.004056 26374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:51.004312 26374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:51.004601 26374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:01:51.010068 26374 server_base.cc:1034] running on GCE node
W20260620 23:01:51.010015 26382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.010015 26380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.010023 26379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.010540 26374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:51.010912 26374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:51.012109 26374 hybrid_clock.cc:648] HybridClock initialized: now 1781996511012110 us; error 48 us; skew 500 ppm
I20260620 23:01:51.013562 26374 webserver.cc:460] Webserver started at http://127.21.66.126:42095/ using document root <none> and password file <none>
I20260620 23:01:51.013823 26374 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:51.013895 26374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:51.014046 26374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:51.015218 26374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "34ed0923e6b0483fa8d5e6c8038a5170"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "7c9f0817cdc7036551606a5d699535c7"
server_key_iv: "d512f4c3d58b35b0b994594d38aacffe"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.015604 26374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "34ed0923e6b0483fa8d5e6c8038a5170"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "7c9f0817cdc7036551606a5d699535c7"
server_key_iv: "d512f4c3d58b35b0b994594d38aacffe"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.017058 26374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:51.017978 26388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.018217 26374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:51.018332 26374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "34ed0923e6b0483fa8d5e6c8038a5170"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "7c9f0817cdc7036551606a5d699535c7"
server_key_iv: "d512f4c3d58b35b0b994594d38aacffe"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.018424 26374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:51.021958 26374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:51.022293 26374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:51.022404 26374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:51.026604 26374 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:34165
I20260620 23:01:51.026675 26440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:34165 every 8 connection(s)
I20260620 23:01:51.027053 26374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:51.027796 26441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:51.030812 26441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170: Bootstrap starting.
I20260620 23:01:51.031414 26441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:51.031728 26441 log.cc:824] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:51.032488 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 26374
I20260620 23:01:51.032625 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:51.032691 26441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170: No bootstrap required, opened a new log
I20260620 23:01:51.032832 21769 external_mini_cluster.cc:1428] Setting key 56b5223de7ed294f7b4a407743bf1fed
I20260620 23:01:51.035300 26441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34165 } }
I20260620 23:01:51.035498 26441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:51.035578 26441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ed0923e6b0483fa8d5e6c8038a5170, State: Initialized, Role: FOLLOWER
I20260620 23:01:51.035730 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.033397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.21.66.126:34165)
0620 23:01:51.033615 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.033622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.033648 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:01:51.034250 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.034258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.034301 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.034423 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.035157 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.035160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.035160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.035164 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.035208 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.035218 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.035281 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.035377 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.035487 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260620 23:01:51.035738 26441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34165 } }
I20260620 23:01:51.035871 26441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:51.035913 26441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:51.035954 26441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:51.037060 26441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34165 } }
I20260620 23:01:51.037153 26441 leader_election.cc:302] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ed0923e6b0483fa8d5e6c8038a5170; no voters:
I20260620 23:01:51.037346 26441 leader_election.cc:288] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:51.037425 26446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:51.037619 26446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [term 1 LEADER]: Becoming Leader. State: Replica: 34ed0923e6b0483fa8d5e6c8038a5170, State: Running, Role: LEADER
I20260620 23:01:51.037664 26441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:51.037716 26446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34165 } }
I20260620 23:01:51.038112 26447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34165 } } }
I20260620 23:01:51.038134 26448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ed0923e6b0483fa8d5e6c8038a5170. Latest consensus state: current_term: 1 leader_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed0923e6b0483fa8d5e6c8038a5170" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34165 } } }
I20260620 23:01:51.038329 26448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:51.038283 26447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:51.038558 26455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:51.039101 26455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:51.040606 26455 catalog_manager.cc:1347] Generated new cluster ID: e4719e7bbee24278ab68eb802de7d06d
I20260620 23:01:51.040650 26455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:51.058847 26455 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:51.059051 26455 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:51.065802 26455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170: Generated new TSK 0
I20260620 23:01:51.066089 26455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:51.074402 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34165
--builtin_ntp_servers=127.21.66.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:01:51.171456 26465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:51.171728 26465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:51.171773 26465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:51.173089 26465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:51.173170 26465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:51.173218 26465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:51.174748 26465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:51.175060 26465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:51.175387 26465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:51.180709 26470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.180733 26473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.181048 26465 server_base.cc:1034] running on GCE node
W20260620 23:01:51.180733 26471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.181413 26465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:51.181751 26465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:51.182955 26465 hybrid_clock.cc:648] HybridClock initialized: now 1781996511182916 us; error 51 us; skew 500 ppm
I20260620 23:01:51.184379 26465 webserver.cc:460] Webserver started at http://127.21.66.65:33461/ using document root <none> and password file <none>
I20260620 23:01:51.184695 26465 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:51.184772 26465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:51.184930 26465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:51.186046 26465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "65fe36103100c900ed67182f4477a30c"
server_key_iv: "527ecb3259e0e17fe6c97a677366f932"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.186532 26465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "65fe36103100c900ed67182f4477a30c"
server_key_iv: "527ecb3259e0e17fe6c97a677366f932"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.188058 26465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:51.188998 26479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.189194 26465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:51.189306 26465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "65fe36103100c900ed67182f4477a30c"
server_key_iv: "527ecb3259e0e17fe6c97a677366f932"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.189419 26465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:51.192934 26465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:51.193226 26465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:51.193380 26465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:51.193631 26465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:51.193961 26465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:51.194016 26465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.194056 26465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:51.194082 26465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.199895 26465 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:43801
I20260620 23:01:51.200047 26592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:43801 every 8 connection(s)
I20260620 23:01:51.200311 26465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:51.203768 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.201111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:56137 (local address 127.21.66.126:34165)
0620 23:01:51.201227 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.201230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.201474 (+ 244us) server_negotiation.cc:407] Connection header received
0620 23:01:51.202051 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.202052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.202059 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.202148 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.203451 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.203452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.203453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.203459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.203476 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.203485 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.203555 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.203663 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.203691 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:01:51.203992 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.201092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34165 (local address 127.21.66.65:56137)
0620 23:01:51.201314 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.201350 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.201883 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.202233 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.202237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.202245 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.203195 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.203195 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.203229 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.203260 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.203568 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.203574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.203578 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.203675 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.203754 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260620 23:01:51.204586 26593 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34165
I20260620 23:01:51.204679 26593 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:51.204797 26593 heartbeater.cc:511] Master 127.21.66.126:34165 requested a full tablet report, sending...
I20260620 23:01:51.205338 26405 ts_manager.cc:194] Registered new tserver with Master: b14f3e6ab5654ef6896ec6ee0bf6041a (127.21.66.65:43801)
I20260620 23:01:51.210598 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 26465
I20260620 23:01:51.210712 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:51.210868 21769 external_mini_cluster.cc:1428] Setting key 4fd41c3a1b2ae32ac74d32056e5d8926
I20260620 23:01:51.211958 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34165
--builtin_ntp_servers=127.21.66.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:01:51.303602 26596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:51.303843 26596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:51.303965 26596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:51.305241 26596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:51.305291 26596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:51.305321 26596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:51.306787 26596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:51.307078 26596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:51.307423 26596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:51.312739 26601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.312757 26604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.312809 26596 server_base.cc:1034] running on GCE node
W20260620 23:01:51.312739 26602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.313318 26596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:51.313659 26596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:51.314833 26596 hybrid_clock.cc:648] HybridClock initialized: now 1781996511314829 us; error 39 us; skew 500 ppm
I20260620 23:01:51.316242 26596 webserver.cc:460] Webserver started at http://127.21.66.66:44633/ using document root <none> and password file <none>
I20260620 23:01:51.316519 26596 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:51.316596 26596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:51.316742 26596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:51.317895 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "65330f41ba644a9e909dd16c8bd6c1f0"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "b5ce6e334f89850c68a9957eca5f532f"
server_key_iv: "8b8ebba9231f52fa2a990037a46add32"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.318301 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "65330f41ba644a9e909dd16c8bd6c1f0"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "b5ce6e334f89850c68a9957eca5f532f"
server_key_iv: "8b8ebba9231f52fa2a990037a46add32"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.319766 26596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:51.320668 26610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.320904 26596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:51.321072 26596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "65330f41ba644a9e909dd16c8bd6c1f0"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "b5ce6e334f89850c68a9957eca5f532f"
server_key_iv: "8b8ebba9231f52fa2a990037a46add32"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.321185 26596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:51.324916 26596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:51.325220 26596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:51.325372 26596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:51.325605 26596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:51.325946 26596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:51.326002 26596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.326048 26596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:51.326081 26596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.332274 26596 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:42615
I20260620 23:01:51.332316 26723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:42615 every 8 connection(s)
I20260620 23:01:51.332682 26596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:01:51.336148 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.333575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:35315 (local address 127.21.66.126:34165)
0620 23:01:51.333662 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.333665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.333880 (+ 215us) server_negotiation.cc:407] Connection header received
0620 23:01:51.334466 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.334468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.334475 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.334584 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.335879 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.335880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.335881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.335885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.335902 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.335906 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.336022 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.336075 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.336089 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:01:51.336431 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.333548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34165 (local address 127.21.66.66:35315)
0620 23:01:51.333748 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.333789 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.334340 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.334645 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.334649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.334657 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.335708 (+ 1051us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.335709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.335742 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.335775 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.335992 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.335997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.336000 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.336090 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.336147 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260620 23:01:51.337249 26724 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34165
I20260620 23:01:51.337374 26724 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:51.337425 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 26596
I20260620 23:01:51.337543 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:51.337564 26724 heartbeater.cc:511] Master 127.21.66.126:34165 requested a full tablet report, sending...
I20260620 23:01:51.337760 21769 external_mini_cluster.cc:1428] Setting key 9fe4441965a3af264283bf54e0757905
I20260620 23:01:51.338033 26405 ts_manager.cc:194] Registered new tserver with Master: 65330f41ba644a9e909dd16c8bd6c1f0 (127.21.66.66:42615)
I20260620 23:01:51.339196 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34165
--builtin_ntp_servers=127.21.66.84:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:01:51.427294 26727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:51.427552 26727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:51.427582 26727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:51.428994 26727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:51.429045 26727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:51.429077 26727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:51.430675 26727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:51.430934 26727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:51.431223 26727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:51.437175 26733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.437180 26732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.437196 26735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.437707 26727 server_base.cc:1034] running on GCE node
I20260620 23:01:51.438097 26727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:51.438489 26727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:51.439708 26727 hybrid_clock.cc:648] HybridClock initialized: now 1781996511439673 us; error 51 us; skew 500 ppm
I20260620 23:01:51.441411 26727 webserver.cc:460] Webserver started at http://127.21.66.67:40553/ using document root <none> and password file <none>
I20260620 23:01:51.441723 26727 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:51.441812 26727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:51.441987 26727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:51.443444 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "03f4b0213fe542c283de6b7a1e9b50eb"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "e098c604c963bcdce79bb97542580c19"
server_key_iv: "b05c3136349e35e4e68e6c4bb118a996"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.443866 26727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "03f4b0213fe542c283de6b7a1e9b50eb"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "e098c604c963bcdce79bb97542580c19"
server_key_iv: "b05c3136349e35e4e68e6c4bb118a996"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.445297 26727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:51.446223 26741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.446475 26727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:51.446609 26727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "03f4b0213fe542c283de6b7a1e9b50eb"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "e098c604c963bcdce79bb97542580c19"
server_key_iv: "b05c3136349e35e4e68e6c4bb118a996"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.446714 26727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:51.451146 26727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:51.451555 26727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:51.451727 26727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:51.452005 26727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:51.452383 26727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:51.452445 26727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.452482 26727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:51.452512 26727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.459362 26727 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:43639
I20260620 23:01:51.459408 26854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:43639 every 8 connection(s)
I20260620 23:01:51.459824 26727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:01:51.463368 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.460840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:50867 (local address 127.21.66.126:34165)
0620 23:01:51.460905 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.460907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.460978 (+ 71us) server_negotiation.cc:407] Connection header received
0620 23:01:51.461665 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.461667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.461674 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.461738 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.463105 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.463107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.463108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.463113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.463129 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.463134 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.463189 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.463279 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.463295 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:01:51.463667 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.460620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34165 (local address 127.21.66.67:50867)
0620 23:01:51.460841 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.460890 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.461534 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.461827 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.461832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.461846 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.462890 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.462890 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.462924 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.462958 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.463186 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.463191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.463195 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.463292 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.463359 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260620 23:01:51.464262 26855 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34165
I20260620 23:01:51.464357 26855 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:51.464485 26855 heartbeater.cc:511] Master 127.21.66.126:34165 requested a full tablet report, sending...
I20260620 23:01:51.464864 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 26727
I20260620 23:01:51.464915 26404 ts_manager.cc:194] Registered new tserver with Master: 03f4b0213fe542c283de6b7a1e9b50eb (127.21.66.67:43639)
I20260620 23:01:51.464970 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:51.465174 21769 external_mini_cluster.cc:1428] Setting key cab2ec2ee34996f6cdb1935f68722633
I20260620 23:01:51.465902 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:01:51.468485 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.467243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.21.66.126:34165)
0620 23:01:51.467301 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.467303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.467433 (+ 130us) server_negotiation.cc:407] Connection header received
0620 23:01:51.467521 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.467522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.467527 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.467568 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.468272 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.468273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.468274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.468277 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.468290 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.468294 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.468349 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.468404 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.468419 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:01:51.470324 26404 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:51.472038 26404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:51.476531 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.474578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43801 (local address 127.0.0.1:40888)
0620 23:01:51.474790 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.474860 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.474922 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.475376 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.475378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.475380 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.476031 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.476032 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.476048 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.476068 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.476352 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.476361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.476364 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.476418 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.476434 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260620 23:01:51.476660 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.474831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:42615 (local address 127.0.0.1:57640)
0620 23:01:51.475024 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.475034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.475072 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.475742 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.475743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.475746 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.476203 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.476204 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.476210 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.476221 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.476505 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.476505 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.476506 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.476561 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.476576 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":137,"threads_started":1}
I20260620 23:01:51.476806 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.474720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.21.66.65:43801)
0620 23:01:51.475215 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.475220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.475230 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:51.475257 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.475260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.475270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.475589 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.476139 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.476149 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.476149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.476154 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.476197 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.476228 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.476666 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.476692 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.476709 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":86,"threads_started":1}
I20260620 23:01:51.476934 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.475115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.21.66.66:42615)
0620 23:01:51.475587 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.475592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.475603 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:51.475628 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.475630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.475640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.475723 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.476313 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.476323 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.476324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.476329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.476364 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.476393 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.476511 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.476567 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.476690 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":84,"threads_started":1}
I20260620 23:01:51.477110 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.474630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43639 (local address 127.0.0.1:47764)
0620 23:01:51.475023 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.475034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.475111 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.475312 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.475319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.475326 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.476371 (+ 1045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.476371 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.476376 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.476385 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.476842 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.476843 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.476844 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.477031 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.477044 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":95,"threads_started":1}
I20260620 23:01:51.477492 26527 tablet_service.cc:1467] Processing CreateTablet for tablet af71b1dd31c0455ea357f50111b66b74 (DEFAULT_TABLE table=test-table [id=d00463bebdd941b782a9d42516dbd566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:51.477636 26658 tablet_service.cc:1467] Processing CreateTablet for tablet af71b1dd31c0455ea357f50111b66b74 (DEFAULT_TABLE table=test-table [id=d00463bebdd941b782a9d42516dbd566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:51.477909 26527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af71b1dd31c0455ea357f50111b66b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:51.477926 26658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af71b1dd31c0455ea357f50111b66b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:51.480667 26869 tablet_bootstrap.cc:492] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a: Bootstrap starting.
I20260620 23:01:51.480930 26870 tablet_bootstrap.cc:492] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Bootstrap starting.
I20260620 23:01:51.481348 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.474728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.21.66.67:43639)
0620 23:01:51.474936 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.474941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.475089 (+ 148us) server_negotiation.cc:407] Connection header received
0620 23:01:51.475188 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.475197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.475210 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.475309 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.476451 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.476460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.476460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.476465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.476513 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.476540 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.476603 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.477103 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.481224 (+ 4121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260620 23:01:51.481436 26869 tablet_bootstrap.cc:654] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:51.481550 26870 tablet_bootstrap.cc:654] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:51.481810 26869 log.cc:824] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:51.481838 26870 log.cc:824] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:51.482089 26789 tablet_service.cc:1467] Processing CreateTablet for tablet af71b1dd31c0455ea357f50111b66b74 (DEFAULT_TABLE table=test-table [id=d00463bebdd941b782a9d42516dbd566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:51.482510 26789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af71b1dd31c0455ea357f50111b66b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:51.482865 26869 tablet_bootstrap.cc:492] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a: No bootstrap required, opened a new log
I20260620 23:01:51.482946 26869 ts_tablet_manager.cc:1397] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:51.485854 26869 raft_consensus.cc:348] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.486058 26869 raft_consensus.cc:374] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:51.486105 26869 raft_consensus.cc:729] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14f3e6ab5654ef6896ec6ee0bf6041a, State: Initialized, Role: FOLLOWER
I20260620 23:01:51.486274 26869 consensus_queue.cc:260] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.486588 26869 ts_tablet_manager.cc:1428] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 23:01:51.486598 26593 heartbeater.cc:503] Master 127.21.66.126:34165 was elected leader, sending a full tablet report...
I20260620 23:01:51.487100 26870 tablet_bootstrap.cc:492] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: No bootstrap required, opened a new log
I20260620 23:01:51.487208 26870 ts_tablet_manager.cc:1397] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 23:01:51.489616 26870 raft_consensus.cc:348] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.489777 26870 raft_consensus.cc:374] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:51.489802 26870 raft_consensus.cc:729] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65330f41ba644a9e909dd16c8bd6c1f0, State: Initialized, Role: FOLLOWER
I20260620 23:01:51.489953 26870 consensus_queue.cc:260] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.490332 26870 ts_tablet_manager.cc:1428] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:01:51.490355 26724 heartbeater.cc:503] Master 127.21.66.126:34165 was elected leader, sending a full tablet report...
I20260620 23:01:51.490412 26873 tablet_bootstrap.cc:492] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Bootstrap starting.
I20260620 23:01:51.491396 26873 tablet_bootstrap.cc:654] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:51.491772 26873 log.cc:824] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:51.492662 26873 tablet_bootstrap.cc:492] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: No bootstrap required, opened a new log
I20260620 23:01:51.492727 26873 ts_tablet_manager.cc:1397] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:01:51.495157 26873 raft_consensus.cc:348] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.495340 26873 raft_consensus.cc:374] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:51.495365 26873 raft_consensus.cc:729] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f4b0213fe542c283de6b7a1e9b50eb, State: Initialized, Role: FOLLOWER
I20260620 23:01:51.495517 26873 consensus_queue.cc:260] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.495853 26873 ts_tablet_manager.cc:1428] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:01:51.495940 26855 heartbeater.cc:503] Master 127.21.66.126:34165 was elected leader, sending a full tablet report...
I20260620 23:01:51.552595 26875 raft_consensus.cc:482] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:51.552738 26875 raft_consensus.cc:504] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.553212 26875 leader_election.cc:288] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b14f3e6ab5654ef6896ec6ee0bf6041a (127.21.66.65:43801), 03f4b0213fe542c283de6b7a1e9b50eb (127.21.66.67:43639)
I20260620 23:01:51.554608 26726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.553415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43639 (local address 127.21.66.66:50527)
0620 23:01:51.553482 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.553494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.553534 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.553690 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.553691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.553693 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.554203 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.554204 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.554212 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.554227 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.554446 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.554447 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.554448 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.554511 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.554530 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:01:51.554600 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.553462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:50527 (local address 127.21.66.67:43639)
0620 23:01:51.553555 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.553558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.553581 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:01:51.553608 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.553608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.553613 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.553684 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.554320 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.554321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.554321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.554325 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.554356 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.554362 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.554433 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.554501 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.554518 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:01:51.554836 26878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.553437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43801 (local address 127.21.66.66:59433)
0620 23:01:51.553630 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.553639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.553723 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.553859 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.553861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.553863 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:51.554413 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:01:51.554414 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:51.554421 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:01:51.554435 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:51.554645 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:51.554646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:51.554647 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:01:51.554740 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:01:51.554754 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":128,"threads_started":1}
I20260620 23:01:51.554840 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.553493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:59433 (local address 127.21.66.65:43801)
0620 23:01:51.553548 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.553550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.553712 (+ 162us) server_negotiation.cc:407] Connection header received
0620 23:01:51.553795 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.553796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.553802 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.553857 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.554551 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.554552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.554553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.554557 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.554572 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.554577 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.554634 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.554754 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.554771 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:01:51.555042 26809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af71b1dd31c0455ea357f50111b66b74" candidate_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" is_pre_election: true
I20260620 23:01:51.555195 26809 raft_consensus.cc:2454] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65330f41ba644a9e909dd16c8bd6c1f0 in term 0.
I20260620 23:01:51.555263 26547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af71b1dd31c0455ea357f50111b66b74" candidate_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" is_pre_election: true
I20260620 23:01:51.555434 26547 raft_consensus.cc:2454] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65330f41ba644a9e909dd16c8bd6c1f0 in term 0.
I20260620 23:01:51.555428 26614 leader_election.cc:302] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f4b0213fe542c283de6b7a1e9b50eb, 65330f41ba644a9e909dd16c8bd6c1f0; no voters:
I20260620 23:01:51.555568 26875 raft_consensus.cc:2781] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:51.555625 26875 raft_consensus.cc:482] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:51.555660 26875 raft_consensus.cc:3037] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:51.556380 26875 raft_consensus.cc:504] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.556540 26875 leader_election.cc:288] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [CANDIDATE]: Term 1 election: Requested vote from peers b14f3e6ab5654ef6896ec6ee0bf6041a (127.21.66.65:43801), 03f4b0213fe542c283de6b7a1e9b50eb (127.21.66.67:43639)
I20260620 23:01:51.556746 26547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af71b1dd31c0455ea357f50111b66b74" candidate_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a"
I20260620 23:01:51.556746 26809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af71b1dd31c0455ea357f50111b66b74" candidate_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f4b0213fe542c283de6b7a1e9b50eb"
I20260620 23:01:51.556826 26809 raft_consensus.cc:3037] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:51.556826 26547 raft_consensus.cc:3037] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:51.557718 26809 raft_consensus.cc:2454] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65330f41ba644a9e909dd16c8bd6c1f0 in term 1.
I20260620 23:01:51.557718 26547 raft_consensus.cc:2454] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65330f41ba644a9e909dd16c8bd6c1f0 in term 1.
I20260620 23:01:51.557940 26612 leader_election.cc:302] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65330f41ba644a9e909dd16c8bd6c1f0, b14f3e6ab5654ef6896ec6ee0bf6041a; no voters:
I20260620 23:01:51.558074 26875 raft_consensus.cc:2781] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:51.558281 26875 raft_consensus.cc:686] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 1 LEADER]: Becoming Leader. State: Replica: 65330f41ba644a9e909dd16c8bd6c1f0, State: Running, Role: LEADER
I20260620 23:01:51.558393 26875 consensus_queue.cc:237] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } }
I20260620 23:01:51.559302 26404 catalog_manager.cc:5495] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65330f41ba644a9e909dd16c8bd6c1f0 (127.21.66.66). New cstate: current_term: 1 leader_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65330f41ba644a9e909dd16c8bd6c1f0" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42615 } health_report { overall_health: HEALTHY } } }
I20260620 23:01:51.570811 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.569879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.21.66.66:42615)
0620 23:01:51.569936 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.569938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.569947 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:51.569988 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.569989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.570001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.570048 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.570634 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.570635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.570635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.570639 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.570653 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.570657 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.570705 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.570746 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.570757 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:01:51.572948 26809 raft_consensus.cc:1261] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 1 FOLLOWER]: Refusing update from remote peer 65330f41ba644a9e909dd16c8bd6c1f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:51.572949 26547 raft_consensus.cc:1261] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a [term 1 FOLLOWER]: Refusing update from remote peer 65330f41ba644a9e909dd16c8bd6c1f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:51.573343 26879 consensus_queue.cc:1035] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f4b0213fe542c283de6b7a1e9b50eb" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:51.573477 26875 consensus_queue.cc:1035] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14f3e6ab5654ef6896ec6ee0bf6041a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:51.576395 26883 mvcc.cc:205] Tried to move back new op lower bound from 7299057711399309312 to 7299057711343689728. Current Snapshot: MvccSnapshot[applied={T|T < 7299057711399309312}]
I20260620 23:01:51.576789 26882 mvcc.cc:205] Tried to move back new op lower bound from 7299057711399309312 to 7299057711343689728. Current Snapshot: MvccSnapshot[applied={T|T < 7299057711399309312}]
I20260620 23:01:51.577725 26885 mvcc.cc:205] Tried to move back new op lower bound from 7299057711399309312 to 7299057711343689728. Current Snapshot: MvccSnapshot[applied={T|T < 7299057711399309312}]
I20260620 23:01:51.579545 26404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:51.579648 26404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:51.580395 26404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34ed0923e6b0483fa8d5e6c8038a5170: Sending DeleteTablet for 3 replicas of tablet af71b1dd31c0455ea357f50111b66b74
I20260620 23:01:51.580734 26527 tablet_service.cc:1514] Processing DeleteTablet for tablet af71b1dd31c0455ea357f50111b66b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:51 UTC) from {username='slave'} at 127.0.0.1:40888
I20260620 23:01:51.580826 26789 tablet_service.cc:1514] Processing DeleteTablet for tablet af71b1dd31c0455ea357f50111b66b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:51 UTC) from {username='slave'} at 127.0.0.1:47764
I20260620 23:01:51.580976 26658 tablet_service.cc:1514] Processing DeleteTablet for tablet af71b1dd31c0455ea357f50111b66b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:51 UTC) from {username='slave'} at 127.0.0.1:57640
I20260620 23:01:51.581187 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 26465
I20260620 23:01:51.581203 26894 tablet_replica.cc:330] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: stopping tablet replica
I20260620 23:01:51.581211 26892 tablet_replica.cc:330] T af71b1dd31c0455ea357f50111b66b74 P b14f3e6ab5654ef6896ec6ee0bf6041a: stopping tablet replica
I20260620 23:01:51.581815 26894 raft_consensus.cc:2229] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:51.581979 26894 raft_consensus.cc:2258] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:51.582901 26893 tablet_replica.cc:330] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: stopping tablet replica
I20260620 23:01:51.583041 26893 raft_consensus.cc:2229] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:51.583169 26893 raft_consensus.cc:2258] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:51.583607 26894 ts_tablet_manager.cc:1905] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:51.583730 26893 ts_tablet_manager.cc:1905] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:51.585575 26893 ts_tablet_manager.cc:1918] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:51.585649 26893 log.cc:1196] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/af71b1dd31c0455ea357f50111b66b74
I20260620 23:01:51.585677 26894 ts_tablet_manager.cc:1918] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:51.585744 26894 log.cc:1196] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/af71b1dd31c0455ea357f50111b66b74
I20260620 23:01:51.585929 26893 ts_tablet_manager.cc:1939] T af71b1dd31c0455ea357f50111b66b74 P 03f4b0213fe542c283de6b7a1e9b50eb: Deleting consensus metadata
I20260620 23:01:51.586021 26894 ts_tablet_manager.cc:1939] T af71b1dd31c0455ea357f50111b66b74 P 65330f41ba644a9e909dd16c8bd6c1f0: Deleting consensus metadata
I20260620 23:01:51.586454 26392 catalog_manager.cc:4841] TS 65330f41ba644a9e909dd16c8bd6c1f0 (127.21.66.66:42615): tablet af71b1dd31c0455ea357f50111b66b74 (table test-table [id=d00463bebdd941b782a9d42516dbd566]) successfully deleted
I20260620 23:01:51.586551 26392 catalog_manager.cc:4841] TS 03f4b0213fe542c283de6b7a1e9b50eb (127.21.66.67:43639): tablet af71b1dd31c0455ea357f50111b66b74 (table test-table [id=d00463bebdd941b782a9d42516dbd566]) successfully deleted
W20260620 23:01:51.587242 26390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:43801 (error 108)
W20260620 23:01:51.587965 26867 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:51.587679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43801 (local address 127.0.0.1:40900)
0620 23:01:51.587750 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.587764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.587814 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.587910 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43801: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:01:51.588176 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 26596
W20260620 23:01:51.589370 26390 catalog_manager.cc:4584] TS b14f3e6ab5654ef6896ec6ee0bf6041a (127.21.66.65:43801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af71b1dd31c0455ea357f50111b66b74: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43801: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:01:51.593631 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 26727
I20260620 23:01:51.598505 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 26374
2026-06-20T23:01:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (708 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 23:01:51.619215 21769 mini_oidc.cc:152] Starting JWKS server
I20260620 23:01:51.620378 21769 webserver.cc:460] Webserver started at https://127.0.0.1:46189/ using document root <none> and password file <none>
I20260620 23:01:51.620527 21769 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:01:51.620733 21769 webserver.cc:460] Webserver started at http://127.0.0.1:33587/ using document root <none> and password file <none>
2026-06-20T23:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:51Z Disabled control of system clock
I20260620 23:01:51.628939 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44453
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:44481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:44453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46189/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/testchainca.cert with env {}
W20260620 23:01:51.713630 26901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:51.713853 26901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:51.713877 26901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:51.715248 26901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:51.715297 26901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:51.715310 26901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:51.715323 26901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:51.715335 26901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:01:51.715348 26901 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:01:51.715359 26901 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46189/jwks.json
I20260620 23:01:51.716881 26901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:44481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:44453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46189/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:51.717097 26901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:51.717258 26901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:51.722463 26906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.722481 26907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.722524 26901 server_base.cc:1034] running on GCE node
W20260620 23:01:51.722519 26909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.722951 26901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:51.723234 26901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:51.724462 26901 hybrid_clock.cc:648] HybridClock initialized: now 1781996511724450 us; error 34 us; skew 500 ppm
I20260620 23:01:51.725867 26901 webserver.cc:460] Webserver started at http://127.21.66.126:39327/ using document root <none> and password file <none>
I20260620 23:01:51.726251 26901 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:51.726334 26901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:51.726501 26901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:51.727780 26901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "ede0b301bfda43cda8c55d78b29c5c99"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "66bc8a28e9475fad5991962f2292b14c"
server_key_iv: "110ea85071bda326b069a58bbcf0d444"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.728186 26901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "ede0b301bfda43cda8c55d78b29c5c99"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "66bc8a28e9475fad5991962f2292b14c"
server_key_iv: "110ea85071bda326b069a58bbcf0d444"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.729918 26901 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 23:01:51.730903 26915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.731130 26901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:51.731277 26901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "ede0b301bfda43cda8c55d78b29c5c99"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "66bc8a28e9475fad5991962f2292b14c"
server_key_iv: "110ea85071bda326b069a58bbcf0d444"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.731364 26901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:51.756494 26901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:51.756924 26901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:51.757193 26901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:51.761801 26901 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:44453
I20260620 23:01:51.761868 26967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:44453 every 8 connection(s)
I20260620 23:01:51.762331 26901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:51.763487 26968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:51.764017 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 26901
I20260620 23:01:51.764235 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:51.764439 21769 external_mini_cluster.cc:1428] Setting key 4c96a002c36d758773bbbc0508b89b66
I20260620 23:01:51.766872 26968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99: Bootstrap starting.
I20260620 23:01:51.767601 26968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:51.767956 26968 log.cc:824] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:51.768863 26968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99: No bootstrap required, opened a new log
I20260620 23:01:51.771003 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.765066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.21.66.126:44453)
0620 23:01:51.765302 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.765312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.765339 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:01:51.765783 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.765790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.765858 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.765993 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:51.766336 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:51.767397 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:51.768600 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:51.769231 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:51.769340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:51.770188 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:51.770195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:51.770196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:51.770393 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:51.770430 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:51.770448 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:51.770506 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:51.770668 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:01:51.770804 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260620 23:01:51.771584 26968 raft_consensus.cc:348] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede0b301bfda43cda8c55d78b29c5c99" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44453 } }
I20260620 23:01:51.771785 26968 raft_consensus.cc:374] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:51.771836 26968 raft_consensus.cc:729] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede0b301bfda43cda8c55d78b29c5c99, State: Initialized, Role: FOLLOWER
I20260620 23:01:51.771966 26968 consensus_queue.cc:260] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede0b301bfda43cda8c55d78b29c5c99" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44453 } }
I20260620 23:01:51.772061 26968 raft_consensus.cc:388] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:51.772100 26968 raft_consensus.cc:482] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:51.772145 26968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:51.772331 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:51.772881 26968 raft_consensus.cc:504] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede0b301bfda43cda8c55d78b29c5c99" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44453 } }
I20260620 23:01:51.773010 26968 leader_election.cc:302] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ede0b301bfda43cda8c55d78b29c5c99; no voters:
I20260620 23:01:51.773216 26968 leader_election.cc:288] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:51.773300 26973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:51.773476 26973 raft_consensus.cc:686] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [term 1 LEADER]: Becoming Leader. State: Replica: ede0b301bfda43cda8c55d78b29c5c99, State: Running, Role: LEADER
I20260620 23:01:51.773540 26968 sys_catalog.cc:564] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:51.773628 26973 consensus_queue.cc:237] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede0b301bfda43cda8c55d78b29c5c99" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44453 } }
I20260620 23:01:51.773978 26973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ede0b301bfda43cda8c55d78b29c5c99. Latest consensus state: current_term: 1 leader_uuid: "ede0b301bfda43cda8c55d78b29c5c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede0b301bfda43cda8c55d78b29c5c99" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44453 } } }
I20260620 23:01:51.774019 26974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ede0b301bfda43cda8c55d78b29c5c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ede0b301bfda43cda8c55d78b29c5c99" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44453 } } }
I20260620 23:01:51.774161 26973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:51.774407 26974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99 [sys.catalog]: This master's current role is: LEADER
W20260620 23:01:51.775406 26989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:51.775468 26989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:51.775590 26982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:51.775840 26982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:51.777386 26982 catalog_manager.cc:1347] Generated new cluster ID: b5901a2707a24368ad7da6f6997637e5
I20260620 23:01:51.777452 26982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:51.785337 26982 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:51.785909 26982 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:51.790598 26982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ede0b301bfda43cda8c55d78b29c5c99: Generated new TSK 0
I20260620 23:01:51.790792 26982 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:01:51.822575 26998 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 23:01:51.822714 26998 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:01:51.822918 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:51.805327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.21.66.126:44453)
0620 23:01:51.805374 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:51.805377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:51.805530 (+ 153us) server_negotiation.cc:407] Connection header received
0620 23:01:51.805637 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:51.805638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:51.805651 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:51.805736 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:01:51.805938 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:51.806781 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:51.807750 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:51.808082 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:51.808153 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:51.808393 (+ 240us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:01:51.822685 (+ 14292us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:01:51.822842 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 23:01:51.822992 26971 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42550: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 23:01:51.823379 26997 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:51.805242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44453 (local address 127.0.0.1:42550)
0620 23:01:51.805450 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:51.805461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:51.805539 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:51.805793 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:51.805794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:51.805796 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:01:51.805859 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:51.805861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:51.806885 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:51.806886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:51.807630 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:51.807633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:51.808192 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:51.808193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:51.808247 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:51.808270 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:01:51.822909 (+ 14639us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:01:51.823064 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.66.126:44453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.66.126:44453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46189/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 23:01:51.823964 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 26901
2026-06-20T23:01:51Z chronyd exiting
I20260620 23:01:51.860489 21769 test_util.cc:175] -----------------------------------------------
I20260620 23:01:51.860577 21769 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781996487602640-21769-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (242 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 23:01:51.861483 21769 mini_oidc.cc:152] Starting JWKS server
I20260620 23:01:51.862321 21769 webserver.cc:460] Webserver started at https://127.0.0.1:35307/ using document root <none> and password file <none>
I20260620 23:01:51.862399 21769 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:01:51.862586 21769 webserver.cc:460] Webserver started at http://127.0.0.1:41485/ using document root <none> and password file <none>
2026-06-20T23:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:51Z Disabled control of system clock
I20260620 23:01:51.871728 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46323
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:46323
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35307/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/testchainca.cert with env {}
W20260620 23:01:51.970616 27004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:51.970966 27004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:51.971011 27004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:51.972834 27004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:51.972895 27004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:51.972913 27004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:51.972931 27004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:51.972949 27004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:01:51.972966 27004 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:01:51.972990 27004 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35307/jwks.json
I20260620 23:01:51.975256 27004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:46323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35307/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:51.975534 27004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:51.975785 27004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:51.982221 27010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.982225 27012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:51.982528 27009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:51.982770 27004 server_base.cc:1034] running on GCE node
I20260620 23:01:51.983042 27004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:51.983444 27004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:51.984695 27004 hybrid_clock.cc:648] HybridClock initialized: now 1781996511984666 us; error 42 us; skew 500 ppm
I20260620 23:01:51.986330 27004 webserver.cc:460] Webserver started at http://127.21.66.126:42417/ using document root <none> and password file <none>
I20260620 23:01:51.986615 27004 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:51.986692 27004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:51.986840 27004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:51.988026 27004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "b1200f0ad559498ca093dd99a8faa3b4"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "40604a8beae95930f083badebc195a6a"
server_key_iv: "bb4e0324f767d0660d33dd0361701e1d"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.988411 27004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "b1200f0ad559498ca093dd99a8faa3b4"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "40604a8beae95930f083badebc195a6a"
server_key_iv: "bb4e0324f767d0660d33dd0361701e1d"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.989818 27004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:51.990787 27018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:51.990984 27004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:51.991089 27004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "b1200f0ad559498ca093dd99a8faa3b4"
format_stamp: "Formatted at 2026-06-20 23:01:51 on dist-test-slave-qgtk"
server_key: "40604a8beae95930f083badebc195a6a"
server_key_iv: "bb4e0324f767d0660d33dd0361701e1d"
server_key_version: "encryptionkey@0"
I20260620 23:01:51.991191 27004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:52.007845 27004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:52.008198 27004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:52.008360 27004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:52.012598 27004 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:46323
I20260620 23:01:52.012671 27070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:46323 every 8 connection(s)
I20260620 23:01:52.013062 27004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:52.013772 27071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:52.016718 27071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4: Bootstrap starting.
I20260620 23:01:52.017139 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27004
I20260620 23:01:52.017258 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:52.017445 21769 external_mini_cluster.cc:1428] Setting key 6a4a60a1c0c3731adaa990f496337040
I20260620 23:01:52.017522 27071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:52.018301 27071 log.cc:824] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:52.019079 27071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4: No bootstrap required, opened a new log
I20260620 23:01:52.021487 27071 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1200f0ad559498ca093dd99a8faa3b4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46323 } }
I20260620 23:01:52.021641 27071 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:52.021665 27071 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1200f0ad559498ca093dd99a8faa3b4, State: Initialized, Role: FOLLOWER
I20260620 23:01:52.021816 27071 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1200f0ad559498ca093dd99a8faa3b4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46323 } }
I20260620 23:01:52.021880 27071 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:52.021924 27071 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:52.021974 27071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:52.022795 27071 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1200f0ad559498ca093dd99a8faa3b4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46323 } }
I20260620 23:01:52.022922 27071 leader_election.cc:302] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1200f0ad559498ca093dd99a8faa3b4; no voters:
I20260620 23:01:52.023172 27071 leader_election.cc:288] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:52.023248 27076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:52.023465 27076 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [term 1 LEADER]: Becoming Leader. State: Replica: b1200f0ad559498ca093dd99a8faa3b4, State: Running, Role: LEADER
I20260620 23:01:52.023470 27071 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:52.023586 27076 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1200f0ad559498ca093dd99a8faa3b4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46323 } }
I20260620 23:01:52.023990 27077 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1200f0ad559498ca093dd99a8faa3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1200f0ad559498ca093dd99a8faa3b4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46323 } } }
I20260620 23:01:52.024230 27077 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:52.024153 27078 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1200f0ad559498ca093dd99a8faa3b4. Latest consensus state: current_term: 1 leader_uuid: "b1200f0ad559498ca093dd99a8faa3b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1200f0ad559498ca093dd99a8faa3b4" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46323 } } }
I20260620 23:01:52.024535 27078 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:52.024956 27074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:52.017978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54828 (local address 127.21.66.126:46323)
0620 23:01:52.018233 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:52.018243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:52.018282 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:01:52.018977 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:52.018985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:52.019061 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:52.019241 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:52.019533 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.020957 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.021945 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.022595 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.022746 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.023908 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:52.023913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:52.023914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:52.024155 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:52.024208 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:52.024227 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:52.024301 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:52.024613 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:01:52.024728 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":115,"threads_started":1}
W20260620 23:01:52.025408 27092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:52.025496 27092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:52.025604 27085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:52.025792 27085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:52.026213 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:52.027541 27085 catalog_manager.cc:1347] Generated new cluster ID: 9521c5d7a24c4bbbb995a4c83beac7bb
I20260620 23:01:52.027585 27085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:52.048882 27085 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:52.049746 27085 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:52.064345 27085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1200f0ad559498ca093dd99a8faa3b4: Generated new TSK 0
I20260620 23:01:52.064602 27085 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:01:52.096326 27101 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 23:01:52.096421 27101 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:01:52.096765 27074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:52.074570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.21.66.126:46323)
0620 23:01:52.074624 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:52.074627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:52.074745 (+ 118us) server_negotiation.cc:407] Connection header received
0620 23:01:52.074805 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:52.074806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:52.074817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:52.074881 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:01:52.075028 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.075554 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.076721 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.076926 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.076982 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.077172 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:01:52.096511 (+ 19339us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:01:52.096703 (+ 192us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 23:01:52.096849 27074 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 23:01:52.096872 27100 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:52.074470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46323 (local address 127.0.0.1:54834)
0620 23:01:52.074651 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:52.074662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:52.074734 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:52.074900 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:52.074900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:52.074903 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:01:52.074964 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:52.074966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.075648 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.075650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:52.076596 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:52.076599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.077006 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.077007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:52.077068 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.077075 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:01:52.096658 (+ 19583us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:01:52.096778 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.66.126:46323: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35307/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260620 23:01:52.097255 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27004
2026-06-20T23:01:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (274 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 23:01:52.135659 21769 mini_oidc.cc:152] Starting JWKS server
I20260620 23:01:52.136413 21769 webserver.cc:460] Webserver started at https://127.0.0.1:45743/ using document root <none> and password file <none>
I20260620 23:01:52.136490 21769 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:01:52.136670 21769 webserver.cc:460] Webserver started at http://127.0.0.1:33953/ using document root <none> and password file <none>
2026-06-20T23:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:52Z Disabled control of system clock
I20260620 23:01:52.144812 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44251
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:42109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:44251
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45743/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:01:52.229393 27107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:52.229655 27107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:52.229694 27107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:52.231060 27107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:52.231135 27107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:52.231158 27107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:52.231181 27107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:52.231225 27107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:01:52.231253 27107 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:01:52.231278 27107 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45743/jwks.json
I20260620 23:01:52.232936 27107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:44251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45743/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:52.233268 27107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:52.233603 27107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:52.238929 27113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:52.238946 27112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:52.239295 27107 server_base.cc:1034] running on GCE node
W20260620 23:01:52.238922 27115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:52.239645 27107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:52.239984 27107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:52.241145 27107 hybrid_clock.cc:648] HybridClock initialized: now 1781996512241097 us; error 73 us; skew 500 ppm
I20260620 23:01:52.242710 27107 webserver.cc:460] Webserver started at http://127.21.66.126:36641/ using document root <none> and password file <none>
I20260620 23:01:52.242972 27107 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:52.243048 27107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:52.243218 27107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:52.244381 27107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "9b99a49b875f4d19963d3752772cb821"
format_stamp: "Formatted at 2026-06-20 23:01:52 on dist-test-slave-qgtk"
server_key: "d401f76a2a0edf5f4bfae5561c00c169"
server_key_iv: "3276bc26895e5cee4198853542664433"
server_key_version: "encryptionkey@0"
I20260620 23:01:52.244777 27107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "9b99a49b875f4d19963d3752772cb821"
format_stamp: "Formatted at 2026-06-20 23:01:52 on dist-test-slave-qgtk"
server_key: "d401f76a2a0edf5f4bfae5561c00c169"
server_key_iv: "3276bc26895e5cee4198853542664433"
server_key_version: "encryptionkey@0"
I20260620 23:01:52.246341 27107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:01:52.247347 27121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:52.247567 27107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:52.247710 27107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "9b99a49b875f4d19963d3752772cb821"
format_stamp: "Formatted at 2026-06-20 23:01:52 on dist-test-slave-qgtk"
server_key: "d401f76a2a0edf5f4bfae5561c00c169"
server_key_iv: "3276bc26895e5cee4198853542664433"
server_key_version: "encryptionkey@0"
I20260620 23:01:52.247805 27107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:52.260869 27107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:52.261274 27107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:52.261466 27107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:52.265982 27107 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:44251
I20260620 23:01:52.266014 27173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:44251 every 8 connection(s)
I20260620 23:01:52.266472 27107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:52.267356 27174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:52.269734 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27107
I20260620 23:01:52.269874 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:52.270061 21769 external_mini_cluster.cc:1428] Setting key fe2bdd400024f57561d0cf7c362aeb43
I20260620 23:01:52.270406 27174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821: Bootstrap starting.
I20260620 23:01:52.271136 27174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:52.271481 27174 log.cc:824] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:52.272339 27174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821: No bootstrap required, opened a new log
I20260620 23:01:52.274715 27174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99a49b875f4d19963d3752772cb821" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44251 } }
I20260620 23:01:52.274897 27174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:52.274932 27174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b99a49b875f4d19963d3752772cb821, State: Initialized, Role: FOLLOWER
I20260620 23:01:52.275067 27174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99a49b875f4d19963d3752772cb821" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44251 } }
I20260620 23:01:52.275146 27174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:52.275178 27174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:52.275226 27174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:52.275975 27174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99a49b875f4d19963d3752772cb821" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44251 } }
I20260620 23:01:52.276113 27174 leader_election.cc:302] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b99a49b875f4d19963d3752772cb821; no voters:
I20260620 23:01:52.276348 27174 leader_election.cc:288] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:52.276422 27179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:52.276614 27179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [term 1 LEADER]: Becoming Leader. State: Replica: 9b99a49b875f4d19963d3752772cb821, State: Running, Role: LEADER
I20260620 23:01:52.276656 27174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:52.276742 27179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99a49b875f4d19963d3752772cb821" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44251 } }
I20260620 23:01:52.277072 27179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b99a49b875f4d19963d3752772cb821. Latest consensus state: current_term: 1 leader_uuid: "9b99a49b875f4d19963d3752772cb821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99a49b875f4d19963d3752772cb821" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44251 } } }
I20260620 23:01:52.277158 27179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:52.277115 27181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b99a49b875f4d19963d3752772cb821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b99a49b875f4d19963d3752772cb821" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44251 } } }
I20260620 23:01:52.277318 27181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:52.277417 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:52.270667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.21.66.126:44251)
0620 23:01:52.270903 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:52.270913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:52.270947 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:01:52.271596 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:52.271604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:52.271696 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:52.271842 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:52.272109 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.273579 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.274546 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.275170 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.275298 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.276388 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:52.276393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:52.276393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:52.276664 (+ 271us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:52.276712 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:52.276729 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:52.277014 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:52.277049 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 23:01:52.277148 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260620 23:01:52.277952 27186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:52.278417 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:52.278782 27186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:52.280506 27186 catalog_manager.cc:1347] Generated new cluster ID: 84182c5eaee24b1b91752d41e800aa89
I20260620 23:01:52.280567 27186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:52.294684 27186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:52.295316 27186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:52.300118 27186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b99a49b875f4d19963d3752772cb821: Generated new TSK 0
I20260620 23:01:52.300339 27186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:52.316922 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:52.312654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.21.66.126:44251)
0620 23:01:52.312723 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:52.312725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:52.312879 (+ 154us) server_negotiation.cc:407] Connection header received
0620 23:01:52.312969 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:52.312970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:52.312981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:52.313046 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:01:52.313282 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.313895 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.315110 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.315383 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.315440 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.315635 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:01:52.316695 (+ 1060us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:01:52.316830 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 23:01:52.316988 27177 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 23:01:52.317018 27203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:52.312594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44251 (local address 127.0.0.1:33104)
0620 23:01:52.312790 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:52.312802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:52.312889 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:52.313064 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:52.313065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:52.313067 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:01:52.313132 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:52.313134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.313995 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.313996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:52.315011 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:52.315014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.315466 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.315467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:52.315528 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.315535 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:01:52.316805 (+ 1270us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:01:52.316911 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.66.126:44251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45743/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
I20260620 23:01:52.317255 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27107
E20260620 23:01:52.327569 27204 webserver.cc:533] Webserver: sslize failed:
E20260620 23:01:52.327679 27204 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T23:01:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T23:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:52Z Disabled control of system clock
I20260620 23:01:52.363837 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45349
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:45349
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/testchainca.cert with env {}
W20260620 23:01:52.452342 27208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:52.452600 27208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:52.452626 27208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:52.453857 27208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:52.453912 27208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:52.453936 27208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:52.453949 27208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:52.453961 27208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:01:52.453974 27208 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:01:52.453985 27208 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 23:01:52.455703 27208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:45349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:52.455937 27208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:52.456118 27208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:52.462308 27216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:52.462273 27214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:52.462281 27213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:52.462618 27208 server_base.cc:1034] running on GCE node
I20260620 23:01:52.462903 27208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:52.463259 27208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:52.464448 27208 hybrid_clock.cc:648] HybridClock initialized: now 1781996512464426 us; error 38 us; skew 500 ppm
I20260620 23:01:52.466007 27208 webserver.cc:460] Webserver started at http://127.21.66.126:34451/ using document root <none> and password file <none>
I20260620 23:01:52.466296 27208 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:52.466372 27208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:52.466521 27208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:52.467634 27208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "4ed2ca41b9cd4390b637764be9865a3e"
format_stamp: "Formatted at 2026-06-20 23:01:52 on dist-test-slave-qgtk"
server_key: "aa3da42bd20f7734049264d623127196"
server_key_iv: "0738b36cbf00dee59e48e1e487be57a6"
server_key_version: "encryptionkey@0"
I20260620 23:01:52.468057 27208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "4ed2ca41b9cd4390b637764be9865a3e"
format_stamp: "Formatted at 2026-06-20 23:01:52 on dist-test-slave-qgtk"
server_key: "aa3da42bd20f7734049264d623127196"
server_key_iv: "0738b36cbf00dee59e48e1e487be57a6"
server_key_version: "encryptionkey@0"
I20260620 23:01:52.469470 27208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:52.470521 27222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:52.470882 27208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:52.471016 27208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "4ed2ca41b9cd4390b637764be9865a3e"
format_stamp: "Formatted at 2026-06-20 23:01:52 on dist-test-slave-qgtk"
server_key: "aa3da42bd20f7734049264d623127196"
server_key_iv: "0738b36cbf00dee59e48e1e487be57a6"
server_key_version: "encryptionkey@0"
I20260620 23:01:52.471103 27208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:52.489257 27208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:52.489624 27208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:52.489799 27208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:52.494180 27208 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:45349
I20260620 23:01:52.494196 27274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:45349 every 8 connection(s)
I20260620 23:01:52.494626 27208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:52.495692 27275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:52.498512 27275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e: Bootstrap starting.
I20260620 23:01:52.499034 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27208
I20260620 23:01:52.499176 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:52.499317 27275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:52.499403 21769 external_mini_cluster.cc:1428] Setting key 80178e01f8255d1e2eb84efc09385bbc
I20260620 23:01:52.499691 27275 log.cc:824] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:52.500599 27275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e: No bootstrap required, opened a new log
I20260620 23:01:52.503429 27275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2ca41b9cd4390b637764be9865a3e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45349 } }
I20260620 23:01:52.503616 27275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:52.503665 27275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ed2ca41b9cd4390b637764be9865a3e, State: Initialized, Role: FOLLOWER
I20260620 23:01:52.503847 27275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2ca41b9cd4390b637764be9865a3e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45349 } }
I20260620 23:01:52.503971 27275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:52.504019 27275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:52.504070 27275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:52.504889 27275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2ca41b9cd4390b637764be9865a3e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45349 } }
I20260620 23:01:52.505016 27275 leader_election.cc:302] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ed2ca41b9cd4390b637764be9865a3e; no voters:
I20260620 23:01:52.505227 27275 leader_election.cc:288] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:52.505366 27280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:52.505563 27280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [term 1 LEADER]: Becoming Leader. State: Replica: 4ed2ca41b9cd4390b637764be9865a3e, State: Running, Role: LEADER
I20260620 23:01:52.505604 27275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:52.505779 27280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2ca41b9cd4390b637764be9865a3e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45349 } }
I20260620 23:01:52.506341 27281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ed2ca41b9cd4390b637764be9865a3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2ca41b9cd4390b637764be9865a3e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45349 } } }
I20260620 23:01:52.506449 27281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:52.507251 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:52.499920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.21.66.126:45349)
0620 23:01:52.500169 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:52.500177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:52.500228 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:01:52.500906 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:52.500916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:52.501006 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:52.501183 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:52.501414 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.503089 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.504309 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.504945 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.505133 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.505992 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:52.505997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:52.505998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:01:52.506259 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:52.506343 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:01:52.506365 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:52.506548 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:52.506937 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 23:01:52.507037 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260620 23:01:52.507272 27282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ed2ca41b9cd4390b637764be9865a3e. Latest consensus state: current_term: 1 leader_uuid: "4ed2ca41b9cd4390b637764be9865a3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ed2ca41b9cd4390b637764be9865a3e" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45349 } } }
I20260620 23:01:52.507563 27282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e [sys.catalog]: This master's current role is: LEADER
W20260620 23:01:52.507752 27296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:52.507833 27296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:52.508018 27289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:52.508316 27289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:52.508631 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:01:52.510223 27289 catalog_manager.cc:1347] Generated new cluster ID: 16dfc14c096d49e9bfd3353370d80a12
I20260620 23:01:52.510288 27289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:52.527554 27289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:52.528297 27289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:52.540089 27289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ed2ca41b9cd4390b637764be9865a3e: Generated new TSK 0
I20260620 23:01:52.540342 27289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:52.973613 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:52.551837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.21.66.126:45349)
0620 23:01:52.551907 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:52.551910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:52.552035 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:01:52.552119 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:52.552119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:52.552129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:52.552191 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:01:52.552350 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.552878 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.554093 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.554553 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.554628 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.554818 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:01:52.973302 (+418484us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:01:52.973549 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40832: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 23:01:52.973707 27279 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40832: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 23:01:52.973863 27304 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:52.551762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45349 (local address 127.0.0.1:40832)
0620 23:01:52.551954 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:52.551967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:52.552044 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:52.552219 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:52.552220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:52.552222 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:01:52.552285 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:52.552287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.552976 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.552977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:52.553961 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:52.553965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:52.554642 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:52.554643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:52.554710 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:52.554717 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:01:52.973560 (+418843us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:01:52.973689 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.66.126:45349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260620 23:01:52.974229 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27208
2026-06-20T23:01:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (636 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:53 dist-test-slave-qgtk krb5kdc[27316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:53 dist-test-slave-qgtk krb5kdc[27316](info): set up 2 sockets
Jun 20 23:01:53 dist-test-slave-qgtk krb5kdc[27316](info): commencing operation
krb5kdc: starting...
W20260620 23:01:54.222702 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:54 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:01:54.354677 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:38521
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:38521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:54.440181 27329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:54.440428 27329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:54.440454 27329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:54.441838 27329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:54.441910 27329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:54.441926 27329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:54.441938 27329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:54.441951 27329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:54.443686 27329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:38521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:54.444013 27329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:54.444281 27329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:54.449843 27337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:54.449931 27329 server_base.cc:1034] running on GCE node
W20260620 23:01:54.449941 27334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:54.449966 27335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:54.450709 27329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:54.451020 27329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:54.452188 27329 hybrid_clock.cc:648] HybridClock initialized: now 1781996514452175 us; error 39 us; skew 500 ppm
Jun 20 23:01:54 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996514, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:54.455413 27329 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:01:54.456055 27329 webserver.cc:460] Webserver started at http://127.21.66.126:37749/ using document root <none> and password file <none>
I20260620 23:01:54.456319 27329 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:54.456390 27329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:54.456544 27329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:54.457669 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "35f85702bf191cff29c3d613ab8f5711"
server_key_iv: "f2ab2fe65733736623815f4ec92e9c19"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.458052 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "35f85702bf191cff29c3d613ab8f5711"
server_key_iv: "f2ab2fe65733736623815f4ec92e9c19"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.459522 27329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:54.460639 27344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:54.460970 27329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:54.461102 27329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "35f85702bf191cff29c3d613ab8f5711"
server_key_iv: "f2ab2fe65733736623815f4ec92e9c19"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.461228 27329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:54.480953 27329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:54.483523 27329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:54.483708 27329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:54.488514 27329 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:38521
I20260620 23:01:54.488554 27396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:38521 every 8 connection(s)
I20260620 23:01:54.488952 27329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:54.489889 27397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:54.490291 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27329
I20260620 23:01:54.490413 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:54.490612 21769 external_mini_cluster.cc:1428] Setting key 1fd27d28953336d503e9fc3981a57d3b
I20260620 23:01:54.492815 27397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8: Bootstrap starting.
I20260620 23:01:54.493497 27397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:54.493806 27397 log.cc:824] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:54.494601 27397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8: No bootstrap required, opened a new log
I20260620 23:01:54.497025 27397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38521 } }
I20260620 23:01:54.497215 27397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:54.497280 27397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d6c2224a48c4f02bd7abaa0d03cdcc8, State: Initialized, Role: FOLLOWER
I20260620 23:01:54.497455 27397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38521 } }
I20260620 23:01:54.497556 27397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:54.497601 27397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:54.497654 27397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:01:54 dist-test-slave-qgtk krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:54.498478 27397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38521 } }
I20260620 23:01:54.498608 27397 leader_election.cc:302] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d6c2224a48c4f02bd7abaa0d03cdcc8; no voters:
I20260620 23:01:54.498805 27397 leader_election.cc:288] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:54.498926 27402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:54.499115 27397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:54.499142 27402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [term 1 LEADER]: Becoming Leader. State: Replica: 7d6c2224a48c4f02bd7abaa0d03cdcc8, State: Running, Role: LEADER
I20260620 23:01:54.499331 27402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38521 } }
I20260620 23:01:54.499704 27403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38521 } } }
I20260620 23:01:54.499781 27403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:54.499938 27404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d6c2224a48c4f02bd7abaa0d03cdcc8. Latest consensus state: current_term: 1 leader_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d6c2224a48c4f02bd7abaa0d03cdcc8" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38521 } } }
I20260620 23:01:54.499994 27404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:54.500216 27408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:54.500808 27408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:54.503338 27408 catalog_manager.cc:1347] Generated new cluster ID: db7b260ad0d04c80ac24fe26028c315c
I20260620 23:01:54.503400 27408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:54.506937 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:54.491249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.21.66.126:38521)
0620 23:01:54.491484 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:54.491496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:54.491531 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:01:54.491961 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:54.491969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:54.492026 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:54.492191 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:54.492865 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.493909 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.494810 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.495432 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.495569 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:54.498374 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:54.498380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:54.498380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:54.498393 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:54.501121 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:54.501538 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:54.501542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:54.501542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:54.501666 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:54.502636 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:54.502637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:54.502637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:54.502968 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:54.503132 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:54.503260 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:01:54.506574 (+ 3314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
I20260620 23:01:54.529259 27408 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:54.530081 27408 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:54.540314 27408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d6c2224a48c4f02bd7abaa0d03cdcc8: Generated new TSK 0
I20260620 23:01:54.540540 27408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:01:54.578606 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:38521
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:54.667634 27423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:54.667905 27423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:54.667949 27423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:54.669183 27423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:54.669253 27423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:54.669299 27423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:54.670969 27423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:54.671263 27423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:54.671520 27423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:54.676896 27431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:54.676896 27429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:54.676975 27428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:54.677788 27423 server_base.cc:1034] running on GCE node
I20260620 23:01:54.678061 27423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:54.678371 27423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:54.679522 27423 hybrid_clock.cc:648] HybridClock initialized: now 1781996514679505 us; error 32 us; skew 500 ppm
Jun 20 23:01:54 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996514, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:54.682399 27423 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:01:54.682880 27423 webserver.cc:460] Webserver started at http://127.21.66.65:40629/ using document root <none> and password file <none>
I20260620 23:01:54.683112 27423 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:54.683182 27423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:54.683334 27423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:54.684464 27423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "01612b07901e4a6991cf645cc120893b"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "5bc763e516a65953a53d7b80cb1f90cb"
server_key_iv: "b609313060a922e99ce0b4a1cfdd6e46"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.684834 27423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "01612b07901e4a6991cf645cc120893b"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "5bc763e516a65953a53d7b80cb1f90cb"
server_key_iv: "b609313060a922e99ce0b4a1cfdd6e46"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.686364 27423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:01:54.687397 27438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:54.687615 27423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:54.687798 27423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "01612b07901e4a6991cf645cc120893b"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "5bc763e516a65953a53d7b80cb1f90cb"
server_key_iv: "b609313060a922e99ce0b4a1cfdd6e46"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.687897 27423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:54.707621 27423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:54.710297 27423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:54.710508 27423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:54.710793 27423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:54.711131 27423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:54.711186 27423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:54.711222 27423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:54.711241 27423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:54.717861 27423 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:45019
I20260620 23:01:54.718247 27551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:45019 every 8 connection(s)
I20260620 23:01:54.718420 27423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:54.725147 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27423
I20260620 23:01:54.725435 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
Jun 20 23:01:54 dist-test-slave-qgtk krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996514, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:54.725677 21769 external_mini_cluster.cc:1428] Setting key 71ed49cf3c8c73798f1751aae135bae1
I20260620 23:01:54.729799 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:54.719273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:53285 (local address 127.21.66.126:38521)
0620 23:01:54.719384 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:54.719386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:54.719577 (+ 191us) server_negotiation.cc:407] Connection header received
0620 23:01:54.720581 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:54.720583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:54.720595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:54.720659 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:54.721356 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.721905 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.723117 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.723340 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.723403 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:54.726273 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:54.726275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:54.726275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:54.726283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:54.728053 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:54.728572 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:54.728573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:54.728573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:54.728645 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:54.728947 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:54.728948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:54.728948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:54.729112 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:54.729182 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:54.729530 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 23:01:54.729639 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:01:54.729842 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:54.719254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:38521 (local address 127.21.66.65:53285)
0620 23:01:54.719427 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:54.719472 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:54.720413 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:54.720735 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:54.720738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:54.721026 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:54.721242 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:54.721246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.722015 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.722016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:54.723026 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:54.723028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.723442 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.723443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:54.723602 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:54.724181 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:54.724189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:54.726171 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:54.728185 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:54.728196 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:54.728207 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:54.728488 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:54.728730 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:54.728731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:54.728732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:54.728868 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:54.729185 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:54.729192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:54.729379 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 23:01:54.729485 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:01:54.729628 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260620 23:01:54.730441 27552 heartbeater.cc:347] Connected to a master server at 127.21.66.126:38521
I20260620 23:01:54.730594 27552 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:54.730821 27552 heartbeater.cc:511] Master 127.21.66.126:38521 requested a full tablet report, sending...
I20260620 23:01:54.731521 27361 ts_manager.cc:194] Registered new tserver with Master: 01612b07901e4a6991cf645cc120893b (127.21.66.65:45019)
I20260620 23:01:54.732585 27361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:53285
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:01:54.753723 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:38521
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:54.838455 27557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:54.838687 27557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:54.838713 27557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:54.839926 27557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:54.839982 27557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:54.840013 27557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:54.841559 27557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:54.841791 27557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:54.842010 27557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:54.847502 27565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:54.847502 27563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:54.847707 27557 server_base.cc:1034] running on GCE node
W20260620 23:01:54.847522 27562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:54.848094 27557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:54.848416 27557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:54.849599 27557 hybrid_clock.cc:648] HybridClock initialized: now 1781996514849583 us; error 33 us; skew 500 ppm
Jun 20 23:01:54 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996514, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:54.852854 27557 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:01:54.853431 27557 webserver.cc:460] Webserver started at http://127.21.66.66:42963/ using document root <none> and password file <none>
I20260620 23:01:54.853708 27557 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:54.853760 27557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:54.853982 27557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:54.855283 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "aecc79b6c4784319aaef1da1cd115995"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "5c636ed8a131616c630cc7ded49ba3d7"
server_key_iv: "d0b372d62483892b60c1467b6cd5de7d"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.855702 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "aecc79b6c4784319aaef1da1cd115995"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "5c636ed8a131616c630cc7ded49ba3d7"
server_key_iv: "d0b372d62483892b60c1467b6cd5de7d"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.857257 27557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:01:54.858381 27572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:54.858637 27557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:54.858772 27557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "aecc79b6c4784319aaef1da1cd115995"
format_stamp: "Formatted at 2026-06-20 23:01:54 on dist-test-slave-qgtk"
server_key: "5c636ed8a131616c630cc7ded49ba3d7"
server_key_iv: "d0b372d62483892b60c1467b6cd5de7d"
server_key_version: "encryptionkey@0"
I20260620 23:01:54.858932 27557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:54.875795 27557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:54.882908 27557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:54.883158 27557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:54.883438 27557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:54.883813 27557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:54.883872 27557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:54.883913 27557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:54.883942 27557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:54.891520 27557 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:39597
I20260620 23:01:54.891559 27685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:39597 every 8 connection(s)
I20260620 23:01:54.892087 27557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:01:54 dist-test-slave-qgtk krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996514, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:54.900305 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27557
I20260620 23:01:54.900436 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:54.900682 21769 external_mini_cluster.cc:1428] Setting key 764944f28b1b4b464926edf4feb189fd
I20260620 23:01:54.903699 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:54.893109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:41929 (local address 127.21.66.126:38521)
0620 23:01:54.893163 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:54.893165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:54.893416 (+ 251us) server_negotiation.cc:407] Connection header received
0620 23:01:54.894908 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:54.894909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:54.894924 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:54.894977 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:54.895677 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.896179 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.897261 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.897466 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.897543 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:54.900077 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:54.900079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:54.900079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:54.900088 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:54.902074 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:54.902603 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:54.902605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:54.902605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:54.902673 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:54.902966 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:54.902967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:54.902967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:54.903099 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:54.903178 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:54.903512 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:01:54.903599 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:54.903924 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:54.893071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:38521 (local address 127.21.66.66:41929)
0620 23:01:54.893283 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:54.893324 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:54.894761 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:54.895073 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:54.895077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:54.895363 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:54.895565 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:54.895570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.896280 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.896281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:54.897149 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:54.897151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:54.897564 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:54.897565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:54.897695 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:54.898305 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:54.898314 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:54.899955 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:54.902219 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:54.902230 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:54.902233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:54.902504 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:54.902752 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:54.902753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:54.902753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:54.902891 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:54.903169 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:54.903175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:54.903328 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 23:01:54.903492 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 23:01:54.903675 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
I20260620 23:01:54.904556 27686 heartbeater.cc:347] Connected to a master server at 127.21.66.126:38521
I20260620 23:01:54.904691 27686 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:54.904951 27686 heartbeater.cc:511] Master 127.21.66.126:38521 requested a full tablet report, sending...
I20260620 23:01:54.905442 27361 ts_manager.cc:194] Registered new tserver with Master: aecc79b6c4784319aaef1da1cd115995 (127.21.66.66:39597)
I20260620 23:01:54.905918 27361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:41929
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:01:54.930552 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:38521
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:55.015301 27691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:55.015553 27691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:55.015597 27691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:55.016850 27691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:55.016927 27691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:55.016975 27691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:55.018678 27691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:55.019007 27691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:55.019270 27691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:55.024856 27696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:55.024829 27699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:55.024870 27691 server_base.cc:1034] running on GCE node
W20260620 23:01:55.024829 27697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:55.025449 27691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:55.025753 27691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:55.026928 27691 hybrid_clock.cc:648] HybridClock initialized: now 1781996515026904 us; error 34 us; skew 500 ppm
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.029717 27691 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:01:55.030335 27691 webserver.cc:460] Webserver started at http://127.21.66.67:46841/ using document root <none> and password file <none>
I20260620 23:01:55.030583 27691 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:55.030632 27691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:55.030805 27691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:55.032346 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "5446356cbb23470fb3a57a00d19308e4"
format_stamp: "Formatted at 2026-06-20 23:01:55 on dist-test-slave-qgtk"
server_key: "ce0d5da2754a0b76b5eaee8ef3dda5d3"
server_key_iv: "d570fe3a301e9ccee1491b0c95f55cd6"
server_key_version: "encryptionkey@0"
I20260620 23:01:55.032756 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "5446356cbb23470fb3a57a00d19308e4"
format_stamp: "Formatted at 2026-06-20 23:01:55 on dist-test-slave-qgtk"
server_key: "ce0d5da2754a0b76b5eaee8ef3dda5d3"
server_key_iv: "d570fe3a301e9ccee1491b0c95f55cd6"
server_key_version: "encryptionkey@0"
I20260620 23:01:55.034232 27691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 23:01:55.035138 27706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:55.035418 27691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:55.035526 27691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "5446356cbb23470fb3a57a00d19308e4"
format_stamp: "Formatted at 2026-06-20 23:01:55 on dist-test-slave-qgtk"
server_key: "ce0d5da2754a0b76b5eaee8ef3dda5d3"
server_key_iv: "d570fe3a301e9ccee1491b0c95f55cd6"
server_key_version: "encryptionkey@0"
I20260620 23:01:55.035607 27691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:55.046727 27691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:55.049194 27691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:55.049356 27691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:55.049636 27691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:55.049965 27691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:55.050019 27691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:55.050061 27691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:55.050089 27691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:55.056493 27691 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:38349
I20260620 23:01:55.056561 27819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:38349 every 8 connection(s)
I20260620 23:01:55.056921 27691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:01:55.065897 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27691
I20260620 23:01:55.066028 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:55.066264 21769 external_mini_cluster.cc:1428] Setting key e42777885f60215c9fc0c4a4d9f78ff9
I20260620 23:01:55.068039 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:55.057705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:49335 (local address 127.21.66.126:38521)
0620 23:01:55.057774 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:55.057777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:55.058142 (+ 365us) server_negotiation.cc:407] Connection header received
0620 23:01:55.059485 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:55.059487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:55.059500 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:55.059550 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:55.060301 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.060844 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.061918 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.062237 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.062321 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:55.064792 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:55.064794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:55.064794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:55.064803 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:55.066561 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.067025 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.067026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.067027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.067087 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.067361 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.067362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.067362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.067484 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:55.067556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:55.067850 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:01:55.067943 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:01:55.068256 27822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:55.057705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:38521 (local address 127.21.66.67:49335)
0620 23:01:55.057890 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:55.057926 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:55.059341 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:55.059636 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:55.059639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:55.059990 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:55.060196 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:55.060201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.060957 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.060958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:55.061802 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:55.061804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.062330 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.062331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:55.062486 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:55.063107 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:55.063116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:55.064672 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:55.066654 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:55.066662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:55.066665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:55.066932 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:55.067156 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:55.067157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:55.067157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:55.067279 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:55.067558 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:55.067566 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:55.067717 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 23:01:55.067830 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:01:55.068010 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260620 23:01:55.068955 27820 heartbeater.cc:347] Connected to a master server at 127.21.66.126:38521
I20260620 23:01:55.069093 27820 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:55.069265 27820 heartbeater.cc:511] Master 127.21.66.126:38521 requested a full tablet report, sending...
I20260620 23:01:55.069693 27361 ts_manager.cc:194] Registered new tserver with Master: 5446356cbb23470fb3a57a00d19308e4 (127.21.66.67:38349)
I20260620 23:01:55.070246 27361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:49335
I20260620 23:01:55.078513 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/test-admin.keytab.
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.100723 21769 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 23:01:55.102752 27827 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:55.101744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.21.66.65:45019)
0620 23:01:55.101933 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:55.101939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:55.101951 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:01:55.101990 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:55.101992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:55.102007 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:55.102107 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:55.102626 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49900: BlockingRecv error: recv got EOF from 127.0.0.1:49900 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
W20260620 23:01:55.102895 21769 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.104138 21769 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.105201 21769 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:55.114565 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:55.105819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.21.66.65:45019)
0620 23:01:55.105883 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:55.105885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:55.105982 (+ 97us) server_negotiation.cc:407] Connection header received
0620 23:01:55.106095 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:55.106096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:55.106110 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:55.106270 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:55.106758 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.107580 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.108414 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.108777 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.108883 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:55.111404 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:55.111421 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:55.111422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:55.111430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:55.113214 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.113658 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.113662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.113663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.113728 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.113945 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.113946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.113947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.114104 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:55.114303 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:55.114362 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 23:01:55.114446 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 23:01:55.116617 27827 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:55.115995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.21.66.65:45019)
0620 23:01:55.116031 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:55.116033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:55.116064 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:01:55.116128 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:55.116129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:55.116138 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:55.116190 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:55.116556 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49920: BlockingRecv error: recv got EOF from 127.0.0.1:49920 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:01:55.116688 21769 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.117755 21769 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.119102 21769 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:55.127480 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:55.119779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.21.66.65:45019)
0620 23:01:55.119954 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:55.119957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:55.119968 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:55.120036 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:55.120037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:55.120049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:55.120125 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:55.120560 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.121069 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.121800 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.122015 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.122106 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:55.124662 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:55.124663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:55.124664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:55.124673 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:55.126256 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.126653 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.126655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.126655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.126718 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.126953 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.126954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.126955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.127068 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:55.127172 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:55.127279 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 23:01:55.127365 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 23:01:55.129654 27827 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:01:55.128854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.21.66.65:45019)
0620 23:01:55.128893 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:55.128895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:55.129110 (+ 215us) server_negotiation.cc:407] Connection header received
0620 23:01:55.129134 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:55.129135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:55.129145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:55.129208 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:55.129593 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49944: BlockingRecv error: recv got EOF from 127.0.0.1:49944 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:01:55.129889 21769 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781996487602640-21769-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.131381 21769 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:55.132573 21769 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:01:55.141466 27827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:55.133349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.21.66.65:45019)
0620 23:01:55.133414 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:55.133416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:55.133498 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:01:55.133620 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:55.133621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:55.133642 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:55.133706 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:55.134194 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.134722 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.135464 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:55.135667 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:55.135770 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:55.138365 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:55.138367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:55.138368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:55.138383 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:55.140121 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.140541 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.140542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.140543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.140613 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:55.140837 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:55.140838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:55.140838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:55.140970 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:55.141083 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:55.141260 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 23:01:55.141351 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:01:55.142413 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27423
I20260620 23:01:55.147575 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27557
I20260620 23:01:55.152119 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27691
I20260620 23:01:55.156817 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27329
2026-06-20T23:01:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2002 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27840](info): set up 2 sockets
Jun 20 23:01:55 dist-test-slave-qgtk krb5kdc[27840](info): commencing operation
krb5kdc: starting...
W20260620 23:01:56.230633 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:01:56 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:01:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab.
Entry for principal oryx/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab.
I20260620 23:01:56.362164 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34795
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab
--principal=oryx/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:56.452297 27853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:56.452584 27853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:56.452629 27853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:56.454052 27853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:56.454180 27853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:56.454214 27853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:56.454252 27853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:56.454283 27853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:56.456080 27853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab
--principal=oryx/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:56.456436 27853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:56.456705 27853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:56.462194 27858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:56.462383 27859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:56.462447 27853 server_base.cc:1034] running on GCE node
W20260620 23:01:56.462217 27861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:56.462775 27853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:56.463099 27853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:56.464293 27853 hybrid_clock.cc:648] HybridClock initialized: now 1781996516464274 us; error 29 us; skew 500 ppm
Jun 20 23:01:56 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:56.467469 27853 init.cc:377] Logged in from keytab as oryx/127.21.66.126@KRBTEST.COM (short username oryx)
I20260620 23:01:56.468008 27853 webserver.cc:460] Webserver started at http://127.21.66.126:40825/ using document root <none> and password file <none>
I20260620 23:01:56.468355 27853 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:56.468430 27853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:56.468580 27853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:56.469660 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "05367f8fccc5489692b9e4debe1b467f"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "046447205ba865d7058f2d8f8e4b3e9b"
server_key_iv: "b1d9b52bc0edeadb7637041bbd748a0e"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.470031 27853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "05367f8fccc5489692b9e4debe1b467f"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "046447205ba865d7058f2d8f8e4b3e9b"
server_key_iv: "b1d9b52bc0edeadb7637041bbd748a0e"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.471542 27853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:56.472668 27868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:56.472884 27853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:56.473017 27853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "05367f8fccc5489692b9e4debe1b467f"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "046447205ba865d7058f2d8f8e4b3e9b"
server_key_iv: "b1d9b52bc0edeadb7637041bbd748a0e"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.473102 27853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:56.492198 27853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:56.494830 27853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:56.494988 27853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:56.499109 27853 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:34795
I20260620 23:01:56.499145 27920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:34795 every 8 connection(s)
I20260620 23:01:56.499573 27853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:01:56.500766 27921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:56.503444 27921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: Bootstrap starting.
I20260620 23:01:56.504109 27921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:56.504441 27921 log.cc:824] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:56.505420 27921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: No bootstrap required, opened a new log
I20260620 23:01:56.507347 27921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } }
I20260620 23:01:56.507539 27921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:56.507596 27921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05367f8fccc5489692b9e4debe1b467f, State: Initialized, Role: FOLLOWER
I20260620 23:01:56.507790 27921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } }
I20260620 23:01:56.507893 27921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:01:56.507942 27921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:01:56.508000 27921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:56.508422 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27853
I20260620 23:01:56.508514 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:01:56.508723 21769 external_mini_cluster.cc:1428] Setting key 2e4e6d0a71824ffd2fa507a5a46114b1
I20260620 23:01:56.509164 27921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } }
I20260620 23:01:56.509326 27921 leader_election.cc:302] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05367f8fccc5489692b9e4debe1b467f; no voters:
I20260620 23:01:56.509527 27921 leader_election.cc:288] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:01:56.509545 27926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:56.509730 27926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 1 LEADER]: Becoming Leader. State: Replica: 05367f8fccc5489692b9e4debe1b467f, State: Running, Role: LEADER
I20260620 23:01:56.509878 27921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:56.509893 27926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } }
I20260620 23:01:56.510337 27928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05367f8fccc5489692b9e4debe1b467f. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } }
I20260620 23:01:56.510408 27928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:56.510599 27930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:56.510918 27927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } }
I20260620 23:01:56.511093 27927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:56.511195 27930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:56.513396 27930 catalog_manager.cc:1347] Generated new cluster ID: daaf26df389f444f9e25258b158619b8
I20260620 23:01:56.513455 27930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:01:56 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:01:56.521636 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:56.509301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.21.66.126:34795)
0620 23:01:56.509525 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:56.509536 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:56.509575 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:01:56.509710 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:56.509719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:56.509785 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:56.509980 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:56.510912 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.512560 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.513726 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.514398 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.514529 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:56.517528 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:56.517533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:56.517534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:56.517547 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:56.519780 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:56.520231 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:56.520234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:56.520234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:56.520374 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:56.520597 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:56.520598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:56.520598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:56.520956 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:56.521112 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:56.521240 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:01:56.521390 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260620 23:01:56.534737 27930 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:01:56.535519 27930 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:56.544656 27930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: Generated new TSK 0
I20260620 23:01:56.544893 27930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab.
Entry for principal oryx/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab.
I20260620 23:01:56.573534 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34795
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab
--principal=oryx/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:56.657205 27947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:56.657433 27947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:56.657459 27947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:56.658708 27947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:56.658752 27947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:56.658780 27947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:56.660291 27947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab
--principal=oryx/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:56.660544 27947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:56.660787 27947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:56.666047 27955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:56.666224 27952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:56.666385 27947 server_base.cc:1034] running on GCE node
W20260620 23:01:56.666461 27953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:56.666666 27947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:56.666944 27947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:56.668121 27947 hybrid_clock.cc:648] HybridClock initialized: now 1781996516668123 us; error 46 us; skew 500 ppm
Jun 20 23:01:56 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:56.670929 27947 init.cc:377] Logged in from keytab as oryx/127.21.66.65@KRBTEST.COM (short username oryx)
I20260620 23:01:56.671471 27947 webserver.cc:460] Webserver started at http://127.21.66.65:33673/ using document root <none> and password file <none>
I20260620 23:01:56.671710 27947 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:56.671774 27947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:56.671916 27947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:56.673018 27947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "bc8d4962f326499fb2dd201cc3c1ef88"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "cb6142e568a6558cb8cf6d3e194baec7"
server_key_iv: "280978210fc9341e6ab32eb2f5d422e6"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.673391 27947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "bc8d4962f326499fb2dd201cc3c1ef88"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "cb6142e568a6558cb8cf6d3e194baec7"
server_key_iv: "280978210fc9341e6ab32eb2f5d422e6"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.674974 27947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:56.675823 27962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:56.676074 27947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:56.676182 27947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "bc8d4962f326499fb2dd201cc3c1ef88"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "cb6142e568a6558cb8cf6d3e194baec7"
server_key_iv: "280978210fc9341e6ab32eb2f5d422e6"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.676262 27947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:56.690907 27947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:56.693275 27947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:56.693396 27947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:56.693594 27947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:56.693928 27947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:56.693964 27947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:56.693987 27947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:56.694031 27947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:56.700152 27947 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:41777
I20260620 23:01:56.700194 28075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:41777 every 8 connection(s)
I20260620 23:01:56.700527 27947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
Jun 20 23:01:56 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:01:56.709125 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 27947
I20260620 23:01:56.709291 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:01:56.709513 21769 external_mini_cluster.cc:1428] Setting key e14b68cf428c7fa692e54714336184ed
I20260620 23:01:56.711822 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:56.701342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:40049 (local address 127.21.66.126:34795)
0620 23:01:56.701470 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:56.701473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:56.701614 (+ 141us) server_negotiation.cc:407] Connection header received
0620 23:01:56.702167 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:56.702169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:56.702184 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:56.702231 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:56.703513 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.704062 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.705097 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.705318 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.705408 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:56.707920 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:56.707922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:56.707922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:56.707931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:56.709786 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:56.710476 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:56.710477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:56.710478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:56.710594 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:56.710902 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:56.710903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:56.710904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:56.711086 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:56.711162 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:56.711540 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 23:01:56.711680 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 23:01:56.712052 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:56.701260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:40049)
0620 23:01:56.701482 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:56.701532 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:56.702046 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:56.702308 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:56.702312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:56.703119 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:56.703323 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:56.703382 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.704170 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.704171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:56.705001 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:56.705003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.705419 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.705420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:56.705545 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:56.706174 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:56.706189 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:56.707780 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:56.709944 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:56.709954 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:56.709957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:56.710374 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:56.710675 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:56.710676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:56.710676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:56.710828 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:56.711183 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:56.711190 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:56.711395 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 23:01:56.711497 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:01:56.711721 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260620 23:01:56.712699 28076 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34795
I20260620 23:01:56.712822 28076 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:56.713003 28076 heartbeater.cc:511] Master 127.21.66.126:34795 requested a full tablet report, sending...
I20260620 23:01:56.713618 27885 ts_manager.cc:194] Registered new tserver with Master: bc8d4962f326499fb2dd201cc3c1ef88 (127.21.66.65:41777)
I20260620 23:01:56.714735 27885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:40049
WARNING: no policy specified for oryx/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab.
Entry for principal oryx/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab.
I20260620 23:01:56.742005 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34795
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab
--principal=oryx/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:56.829252 28081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:56.829474 28081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:56.829500 28081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:56.830713 28081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:56.830758 28081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:56.830793 28081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:56.832325 28081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab
--principal=oryx/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:56.832592 28081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:56.832849 28081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:56.837977 28089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:56.838002 28087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:56.837982 28086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:56.838297 28081 server_base.cc:1034] running on GCE node
I20260620 23:01:56.838536 28081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:56.838842 28081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:56.840006 28081 hybrid_clock.cc:648] HybridClock initialized: now 1781996516839988 us; error 38 us; skew 500 ppm
Jun 20 23:01:56 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:56.842948 28081 init.cc:377] Logged in from keytab as oryx/127.21.66.66@KRBTEST.COM (short username oryx)
I20260620 23:01:56.843436 28081 webserver.cc:460] Webserver started at http://127.21.66.66:40493/ using document root <none> and password file <none>
I20260620 23:01:56.843688 28081 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:56.843737 28081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:56.843870 28081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:56.844975 28081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "ce78aa9ed2f745968b8f29b347691907"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "03ed749f0fb139f2c4fd22aad1910a4d"
server_key_iv: "c00a564a294150ab03d524579f490501"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.845353 28081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "ce78aa9ed2f745968b8f29b347691907"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "03ed749f0fb139f2c4fd22aad1910a4d"
server_key_iv: "c00a564a294150ab03d524579f490501"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.846814 28081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:01:56.847854 28096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:56.848152 28081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:56.848348 28081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "ce78aa9ed2f745968b8f29b347691907"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "03ed749f0fb139f2c4fd22aad1910a4d"
server_key_iv: "c00a564a294150ab03d524579f490501"
server_key_version: "encryptionkey@0"
I20260620 23:01:56.848424 28081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:56.873580 28081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:56.876256 28081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:56.876497 28081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:56.876760 28081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:56.877116 28081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:56.877151 28081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:56.877202 28081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:56.877238 28081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:56.883816 28081 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:36603
I20260620 23:01:56.883898 28209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:36603 every 8 connection(s)
I20260620 23:01:56.884202 28081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:01:56.887929 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 28081
I20260620 23:01:56.888017 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:01:56.888224 21769 external_mini_cluster.cc:1428] Setting key 29c75eb5259b13d8eed70880fbbb2067
Jun 20 23:01:56 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:01:56.896023 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:56.885251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:49195 (local address 127.21.66.126:34795)
0620 23:01:56.885433 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:56.885435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:56.885619 (+ 184us) server_negotiation.cc:407] Connection header received
0620 23:01:56.886185 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:56.886187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:56.886200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:56.886279 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:56.887233 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.887848 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.889426 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.889654 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.889805 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:56.892474 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:56.892476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:56.892477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:56.892485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:56.894207 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:56.894876 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:56.894878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:56.894878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:56.894949 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:56.895242 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:56.895244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:56.895244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:56.895381 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:56.895479 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:56.895801 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:01:56.895906 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 23:01:56.896287 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:56.885174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.66:49195)
0620 23:01:56.885481 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:56.885522 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:56.886033 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:56.886339 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:56.886342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:56.886932 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:56.887130 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:56.887135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.887981 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.887985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:56.889318 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:56.889320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:56.889817 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:56.889818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:56.889975 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:56.890639 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:56.890649 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:56.892363 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:56.894453 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:56.894461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:56.894463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:56.894739 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:56.895029 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:56.895030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:56.895031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:56.895161 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:56.895476 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:56.895485 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:56.895653 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 23:01:56.895779 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 23:01:56.896022 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":204,"threads_started":1}
WARNING: no policy specified for oryx/127.21.66.67@KRBTEST.COM; defaulting to no policy
I20260620 23:01:56.896852 28210 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34795
I20260620 23:01:56.896988 28210 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:56.897219 28210 heartbeater.cc:511] Master 127.21.66.126:34795 requested a full tablet report, sending...
I20260620 23:01:56.897656 27885 ts_manager.cc:194] Registered new tserver with Master: ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:01:56.898283 27885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:49195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab.
Entry for principal oryx/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab.
I20260620 23:01:56.917431 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34795
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab
--principal=oryx/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:57.001060 28215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:57.001304 28215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:57.001332 28215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:57.002660 28215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:57.002704 28215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:57.002733 28215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:57.004200 28215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab
--principal=oryx/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:57.004419 28215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:57.004601 28215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:57.010388 28221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.010537 28215 server_base.cc:1034] running on GCE node
W20260620 23:01:57.010673 28220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.010659 28223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.010962 28215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:57.011256 28215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:57.012449 28215 hybrid_clock.cc:648] HybridClock initialized: now 1781996517012428 us; error 38 us; skew 500 ppm
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:57.015794 28215 init.cc:377] Logged in from keytab as oryx/127.21.66.67@KRBTEST.COM (short username oryx)
I20260620 23:01:57.016367 28215 webserver.cc:460] Webserver started at http://127.21.66.67:45045/ using document root <none> and password file <none>
I20260620 23:01:57.016649 28215 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.016729 28215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.016891 28215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:57.018083 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "d2bbd190c5d24aeea1c0eb118d633f6a"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "df5bc0554fc845414f6ec5f53f2d5233"
server_key_iv: "3369bd8872b80382460e3c5e3c5b1dae"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.018571 28215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "d2bbd190c5d24aeea1c0eb118d633f6a"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "df5bc0554fc845414f6ec5f53f2d5233"
server_key_iv: "3369bd8872b80382460e3c5e3c5b1dae"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.020088 28215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:01:57.021175 28230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.021507 28215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:57.021646 28215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "d2bbd190c5d24aeea1c0eb118d633f6a"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "df5bc0554fc845414f6ec5f53f2d5233"
server_key_iv: "3369bd8872b80382460e3c5e3c5b1dae"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.021747 28215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.040544 28215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:57.043553 28215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:57.043761 28215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:57.044070 28215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:57.044471 28215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:57.044534 28215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.044581 28215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:57.044615 28215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.051482 28215 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:38565
I20260620 23:01:57.051522 28343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:38565 every 8 connection(s)
I20260620 23:01:57.051959 28215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:01:57.053076 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 28215
I20260620 23:01:57.053186 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:01:57.053395 21769 external_mini_cluster.cc:1428] Setting key f571ea7f65e26f6b6544efdf15077819
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:01:57.062916 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.052999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:45943 (local address 127.21.66.126:34795)
0620 23:01:57.053063 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.053065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.053416 (+ 351us) server_negotiation.cc:407] Connection header received
0620 23:01:57.054196 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.054197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.054211 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.054263 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.055382 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.055954 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.057063 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.057268 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.057333 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.059696 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.059702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.059702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.059711 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.061338 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.061831 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.061832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.061833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.061891 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.062239 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.062240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.062241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.062364 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.062429 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.062760 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.062844 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:01:57.063025 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.052995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.67:45943)
0620 23:01:57.053191 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.053246 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.054070 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.054355 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.054360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.055055 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.055280 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.055285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.056054 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.056055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.056969 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.056972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.057413 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.057414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.057529 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.058153 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.058164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.059593 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.061461 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.061470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.061473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.061752 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.061982 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.061983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.061983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.062171 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.062468 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.062474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.062611 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.062704 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.062848 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260620 23:01:57.063617 28344 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34795
I20260620 23:01:57.063709 28344 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:57.063951 28344 heartbeater.cc:511] Master 127.21.66.126:34795 requested a full tablet report, sending...
I20260620 23:01:57.064497 27885 ts_manager.cc:194] Registered new tserver with Master: d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565)
I20260620 23:01:57.064966 27885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:45943
I20260620 23:01:57.065454 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.66.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.125.keytab.
Entry for principal oryx/127.21.66.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.125.keytab.
I20260620 23:01:57.093647 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:44163
--webserver_interface=127.21.66.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.125.keytab
--principal=oryx/127.21.66.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34795,127.21.66.125:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:57.183002 28349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:57.183229 28349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:57.183254 28349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:57.184438 28349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:57.184486 28349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:57.184501 28349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:57.184513 28349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:57.184526 28349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:57.186012 28349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34795,127.21.66.125:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.125.keytab
--principal=oryx/127.21.66.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.66.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
W20260620 23:01:57.186194 28349 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.66.126:34795,127.21.66.125:44163'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 23:01:57.186479 28349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:57.186748 28349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:01:57.191987 28349 server_base.cc:1034] running on GCE node
W20260620 23:01:57.192081 28357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.192147 28355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.192312 28354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.192674 28349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:57.193028 28349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:57.194198 28349 hybrid_clock.cc:648] HybridClock initialized: now 1781996517194177 us; error 41 us; skew 500 ppm
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:57.197160 28349 init.cc:377] Logged in from keytab as oryx/127.21.66.125@KRBTEST.COM (short username oryx)
I20260620 23:01:57.197737 28349 webserver.cc:460] Webserver started at http://127.21.66.125:45513/ using document root <none> and password file <none>
I20260620 23:01:57.197990 28349 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.198061 28349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.198233 28349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:57.199407 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/instance:
uuid: "7ca6226bb5854c73a510b88457ae8789"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4cc96f07a5299c5ada1a3a150e3dfb6f"
server_key_iv: "6c16efb8aabf6d0ed520a295d205c366"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.199787 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal/instance:
uuid: "7ca6226bb5854c73a510b88457ae8789"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4cc96f07a5299c5ada1a3a150e3dfb6f"
server_key_iv: "6c16efb8aabf6d0ed520a295d205c366"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.201308 28349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:01:57.202490 28364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.202752 28349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:57.202901 28349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
uuid: "7ca6226bb5854c73a510b88457ae8789"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4cc96f07a5299c5ada1a3a150e3dfb6f"
server_key_iv: "6c16efb8aabf6d0ed520a295d205c366"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.202991 28349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.216950 28349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:57.219409 28349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:57.219583 28349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:57.223946 28349 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.125:44163
I20260620 23:01:57.224015 28416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.125:44163 every 8 connection(s)
I20260620 23:01:57.224433 28349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
I20260620 23:01:57.225508 28417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:57.228108 28417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:57.229321 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 28349
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.125@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.66.125@KRBTEST.COM
I20260620 23:01:57.240106 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.229001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.21.66.126:34795)
0620 23:01:57.229061 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.229064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.229352 (+ 288us) server_negotiation.cc:407] Connection header received
0620 23:01:57.230082 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.230084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.230096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.230169 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.231058 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.231846 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.233286 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.233593 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.233682 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.236578 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.236579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.236580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.236587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.238276 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.238907 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.238908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.238908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.238989 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.239304 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.239305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.239305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.239503 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.239563 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.239950 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.240036 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:01:57.240404 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.228978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41218)
0620 23:01:57.229193 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.229248 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.229963 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.230263 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.230267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.230664 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.230954 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.230958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.231974 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.231975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.233192 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.233195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.233695 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.233695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.233832 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.234578 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.234589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.236466 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.238405 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.238411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.238413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.238821 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.239078 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.239079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.239080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.239213 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.239581 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.239586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.239759 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.239898 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.240120 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260620 23:01:57.241178 28417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:01:57.241669 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.230048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.21.66.125:44163)
0620 23:01:57.231285 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.231290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.231311 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:01:57.231330 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.231345 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.231363 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.231427 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.232210 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.233529 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.234630 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.234939 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.234999 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.238064 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.238072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.238072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.238078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.240199 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.240608 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.240611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.240611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.240674 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.240892 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.240893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.240894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.241103 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.241264 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.241432 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.241522 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":96,"threads_started":1}
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.125@KRBTEST.COM for oryx/127.21.66.125@KRBTEST.COM
I20260620 23:01:57.249167 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.241526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.21.66.125:44163)
0620 23:01:57.241760 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.241762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.241771 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:57.241794 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.241795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.241807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.242203 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.242230 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.242755 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.243565 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.243772 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.243829 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.245832 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.245833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.245833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.245841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.247955 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.248304 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.248305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.248305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.248436 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.248677 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.248678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.248678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.248804 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.248870 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.248999 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.249073 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260620 23:01:57.249284 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.241482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.0.0.1:34978)
0620 23:01:57.241635 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.241645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.241684 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.241898 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.241899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.242046 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.242110 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.242139 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.242823 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.242824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.243483 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.243485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.243864 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.243865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.243931 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.244355 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.244361 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.245730 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.248049 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.248050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.248050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.248235 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.248524 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.248525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.248525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.248589 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.248883 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.248884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.248925 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.248984 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.249143 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":74}
I20260620 23:01:57.250932 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Bootstrap starting.
I20260620 23:01:57.251688 28417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:57.251983 28417 log.cc:824] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:57.252947 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: No bootstrap required, opened a new log
I20260620 23:01:57.254769 28417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } }
I20260620 23:01:57.254938 28417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:57.255012 28417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca6226bb5854c73a510b88457ae8789, State: Initialized, Role: FOLLOWER
I20260620 23:01:57.255178 28417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } }
I20260620 23:01:57.255542 28417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:57.255525 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } } }
I20260620 23:01:57.255668 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:01:57.257328 28435 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:57.257393 28435 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:57.259451 28349 master_runner.cc:428] Detected that this master 7ca6226bb5854c73a510b88457ae8789 is joining an existing cluster
I20260620 23:01:57.259495 28349 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.66.125:44163
I20260620 23:01:57.267379 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.260583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.21.66.125:44163)
0620 23:01:57.260681 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.260684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.260984 (+ 300us) server_negotiation.cc:407] Connection header received
0620 23:01:57.261095 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.261095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.261104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.261157 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.261517 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.262283 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.263012 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.263237 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.263326 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.264673 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.264674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.264675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.264681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.266228 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.266581 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.266582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.266583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.266632 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.266832 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.266833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.266833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.266952 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.267025 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.267151 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.267252 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":79}
I20260620 23:01:57.267633 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.260494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.0.0.1:34986)
0620 23:01:57.260934 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.260944 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.261011 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.261161 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.261162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.261364 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.261419 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.261438 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.262380 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.262381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.262933 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.262934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.263345 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.263346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.263410 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.264041 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.264046 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.264579 (+ 533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.266318 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.266319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.266319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.266494 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.266702 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.266702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.266703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.266758 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.267016 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.267017 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.267055 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.267124 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.267378 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":83,"thread_start_us":104,"threads_started":1}
I20260620 23:01:57.268980 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.260635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.21.66.126:34795)
0620 23:01:57.260681 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.260684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.260804 (+ 120us) server_negotiation.cc:407] Connection header received
0620 23:01:57.260892 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.260893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.260906 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.260967 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.261466 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.262289 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.263482 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.263739 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.263819 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.264767 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.264768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.264769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.264778 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.267105 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.268087 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.268088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.268089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.268157 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.268473 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.268473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.268474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.268589 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.268645 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.268858 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.268919 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:57.269057 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.260480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41232)
0620 23:01:57.260738 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.260747 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.260820 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.260997 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.260998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.261286 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.261376 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.261378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.262365 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.262366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.263395 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.263398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.263818 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.263818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.263900 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.264265 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.264269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.264671 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.267214 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.267216 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.267216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.267964 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.268239 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.268240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.268240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.268312 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.268672 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.268674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.268738 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.268842 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.268945 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.126@KRBTEST.COM for oryx/127.21.66.125@KRBTEST.COM
I20260620 23:01:57.279677 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.271574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.21.66.125:44163)
0620 23:01:57.271621 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.271623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.271774 (+ 151us) server_negotiation.cc:407] Connection header received
0620 23:01:57.271837 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.271838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.271850 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.271918 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.272357 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.272834 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.273647 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.273845 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.273929 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.276471 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.276473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.276474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.276482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.278165 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.278659 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.278661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.278661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.278726 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.278970 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.278971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.278971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.279136 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.279220 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.279439 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.279535 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:01:57.279670 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.271506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.0.0.1:34994)
0620 23:01:57.271660 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.271704 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.271777 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.271924 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.271927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.272188 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.272238 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.272240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.272933 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.272934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.273534 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.273548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.273932 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.273933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.274033 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.274652 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.274657 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.276371 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.278245 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.278255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.278258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.278558 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.278797 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.278798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.278798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.278873 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.279210 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.279218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.279280 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.279394 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.279545 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":106,"threads_started":1}
I20260620 23:01:57.280397 27885 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.66.125:44163: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7ca6226bb5854c73a510b88457ae8789"
member_type: NON_VOTER
last_known_addr {
host: "127.21.66.125"
port: 44163
}
attrs {
promote: true
}
}
dest_uuid: "05367f8fccc5489692b9e4debe1b467f"
cas_config_opid_index: -1
I20260620 23:01:57.280755 27885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } }
I20260620 23:01:57.281311 28445 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7ca6226bb5854c73a510b88457ae8789 (127.21.66.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } }
I20260620 23:01:57.281713 28391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:57.282481 28445 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.66.125:44163
I20260620 23:01:57.282581 28445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } }
I20260620 23:01:57.282655 28445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:57.282737 28391 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Refusing update from remote peer 05367f8fccc5489692b9e4debe1b467f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 23:01:57.283020 28349 master.cc:537] Master@127.21.66.125:44163 shutting down...
I20260620 23:01:57.283233 28444 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:01:57.284492 27872 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 23:01:57.285027 27872 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f -> Peer 7ca6226bb5854c73a510b88457ae8789 (127.21.66.125:44163): Couldn't send request to peer 7ca6226bb5854c73a510b88457ae8789. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:01:57.286094 28349 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:57.286226 28349 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:57.286284 28349 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: stopping tablet replica
I20260620 23:01:57.299681 28349 master.cc:552] Master@127.21.66.125:44163 shutdown complete.
I20260620 23:01:57.299762 28349 master_runner.cc:305] Clearing existing system tablet
I20260620 23:01:57.299919 28349 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.299979 28349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.300853 28349 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 23:01:57.301610 28449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.301808 28349 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.301908 28349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:57.301929 28349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
uuid: "7ca6226bb5854c73a510b88457ae8789"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4cc96f07a5299c5ada1a3a150e3dfb6f"
server_key_iv: "6c16efb8aabf6d0ed520a295d205c366"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.302709 28349 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:57.304174 28349 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:01:57.304250 28349 log.cc:1196] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 23:01:57.304546 28349 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Deleting consensus metadata
I20260620 23:01:57.304695 28349 master_runner.cc:315] Copying system tablet from 127.21.66.126:34795
I20260620 23:01:57.305274 28349 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: tablet copy: Beginning tablet copy session from remote peer at address 127.21.66.126:34795
I20260620 23:01:57.312263 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.305703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.21.66.126:34795)
0620 23:01:57.305780 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.305782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.305916 (+ 134us) server_negotiation.cc:407] Connection header received
0620 23:01:57.306040 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.306041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.306051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.306131 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.306521 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.307026 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.307873 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.308091 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.308181 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.309219 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.309221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.309222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.309230 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.310885 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.311422 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.311424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.311424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.311498 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.311704 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.311705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.311705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.311833 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.311911 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.312064 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.312166 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:01:57.312279 28455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.305640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41244)
0620 23:01:57.305838 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.305848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.305933 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.306201 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.306202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.306365 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.306424 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.306425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.307171 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.307172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.307776 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.307778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.308183 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.308184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.308255 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.308795 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.308800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.309119 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.311036 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.311038 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.311038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.311315 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.311569 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.311570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.311571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.311634 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.311902 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.311903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.311952 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.312037 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.312157 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260620 23:01:57.312619 27905 tablet_copy_service.cc:140] P 05367f8fccc5489692b9e4debe1b467f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7ca6226bb5854c73a510b88457ae8789 ({username='oryx', principal='oryx/127.21.66.125@KRBTEST.COM'} at 127.0.0.1:41244)
I20260620 23:01:57.312721 27905 tablet_copy_service.cc:161] P 05367f8fccc5489692b9e4debe1b467f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7ca6226bb5854c73a510b88457ae8789 at {username='oryx', principal='oryx/127.21.66.125@KRBTEST.COM'} at 127.0.0.1:41244: session id = 7ca6226bb5854c73a510b88457ae8789-00000000000000000000000000000000
I20260620 23:01:57.313371 27905 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:01:57.314009 28349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:01:57.315053 28349 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: tablet copy: Starting download of 0 data blocks...
I20260620 23:01:57.315207 28349 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: tablet copy: Starting download of 1 WAL segments...
I20260620 23:01:57.316156 28349 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:01:57.317109 27905 tablet_copy_service.cc:342] P 05367f8fccc5489692b9e4debe1b467f: Request end of tablet copy session 7ca6226bb5854c73a510b88457ae8789-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.66.125@KRBTEST.COM'} at 127.0.0.1:41244
I20260620 23:01:57.317184 27905 tablet_copy_service.cc:433] P 05367f8fccc5489692b9e4debe1b467f: ending tablet copy session 7ca6226bb5854c73a510b88457ae8789-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7ca6226bb5854c73a510b88457ae8789
I20260620 23:01:57.319996 28349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:57.321203 28457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.321271 28456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.321271 28459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.323390 28349 server_base.cc:1034] running on GCE node
I20260620 23:01:57.323509 28349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:57.323781 28349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:57.324946 28349 hybrid_clock.cc:648] HybridClock initialized: now 1781996517324929 us; error 32 us; skew 500 ppm
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:57.326712 28349 init.cc:377] Logged in from keytab as oryx/127.21.66.125@KRBTEST.COM (short username oryx)
I20260620 23:01:57.327071 28349 webserver.cc:460] Webserver started at http://127.21.66.125:33245/ using document root <none> and password file <none>
I20260620 23:01:57.327196 28349 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.327226 28349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.327922 28349 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.328534 28466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.328670 28349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.328722 28349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
uuid: "7ca6226bb5854c73a510b88457ae8789"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4cc96f07a5299c5ada1a3a150e3dfb6f"
server_key_iv: "6c16efb8aabf6d0ed520a295d205c366"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.328821 28349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.341589 28349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:57.343863 28349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:57.347625 28349 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.125:44163
I20260620 23:01:57.347972 28518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.125:44163 every 8 connection(s)
I20260620 23:01:57.348160 28349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
I20260620 23:01:57.348944 28519 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:01:57.349412 28519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Bootstrap starting.
I20260620 23:01:57.351881 28519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:01:57.352121 28519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Bootstrap complete.
I20260620 23:01:57.352330 28519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } }
I20260620 23:01:57.352401 28519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7ca6226bb5854c73a510b88457ae8789, State: Initialized, Role: LEARNER
I20260620 23:01:57.352429 28519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } }
I20260620 23:01:57.352635 28519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:57.352628 28522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } }
I20260620 23:01:57.352775 28522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:01:57.354174 28533 catalog_manager.cc:1259] Loaded cluster ID: daaf26df389f444f9e25258b158619b8
I20260620 23:01:57.354254 28533 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: loading cluster ID for follower catalog manager: success
I20260620 23:01:57.355405 28533 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: acquiring CA information for follower catalog manager: success
I20260620 23:01:57.355708 28533 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:01:57.402246 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.394203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.21.66.125:44163)
0620 23:01:57.394374 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.394376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.394388 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:01:57.394525 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.394526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.394541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.394643 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.395268 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.396058 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.397065 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.397306 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.397400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.399001 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.399003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.399003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.399019 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.400661 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.401110 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.401112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.401113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.401182 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.401492 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.401493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.401493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.401697 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.401841 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.401971 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.402069 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for oryx/127.21.66.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.124.keytab.
Entry for principal oryx/127.21.66.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.124.keytab.
I20260620 23:01:57.430433 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:46137
--webserver_interface=127.21.66.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.124.keytab
--principal=oryx/127.21.66.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:57.514320 28537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:57.514580 28537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:57.514606 28537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:57.515930 28537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:01:57.515978 28537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:57.515993 28537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:57.516007 28537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:01:57.516019 28537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:01:57.517647 28537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.124.keytab
--principal=oryx/127.21.66.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:46137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.66.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:57.517966 28537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:57.518265 28537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:57.523384 28543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.523481 28542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.523376 28545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.523931 28537 server_base.cc:1034] running on GCE node
I20260620 23:01:57.524195 28537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:57.524542 28537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:57.525729 28537 hybrid_clock.cc:648] HybridClock initialized: now 1781996517525696 us; error 44 us; skew 500 ppm
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:57.528636 28537 init.cc:377] Logged in from keytab as oryx/127.21.66.124@KRBTEST.COM (short username oryx)
I20260620 23:01:57.529156 28537 webserver.cc:460] Webserver started at http://127.21.66.124:37653/ using document root <none> and password file <none>
I20260620 23:01:57.529384 28537 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.529451 28537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.529616 28537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:01:57.530820 28537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/instance:
uuid: "cfd59e730385467685ff491c0ff20e3c"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4c3bd9a1dfebadfa2f526a7e86cd51b9"
server_key_iv: "eeab5969f4155feab8218fdc652dd311"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.531183 28537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal/instance:
uuid: "cfd59e730385467685ff491c0ff20e3c"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4c3bd9a1dfebadfa2f526a7e86cd51b9"
server_key_iv: "eeab5969f4155feab8218fdc652dd311"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.532639 28537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:57.533465 28552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.533742 28537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:57.533839 28537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
uuid: "cfd59e730385467685ff491c0ff20e3c"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4c3bd9a1dfebadfa2f526a7e86cd51b9"
server_key_iv: "eeab5969f4155feab8218fdc652dd311"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.533936 28537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.546003 28537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:57.548106 28537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:57.548285 28537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:57.553040 28537 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.124:46137
I20260620 23:01:57.553083 28604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.124:46137 every 8 connection(s)
I20260620 23:01:57.553457 28537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
I20260620 23:01:57.554702 28605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:57.555982 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 28537
I20260620 23:01:57.556421 28605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.124@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.66.124@KRBTEST.COM
I20260620 23:01:57.566067 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.557375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.21.66.126:34795)
0620 23:01:57.557529 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.557531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.557544 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:01:57.557760 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.557762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.557774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.557833 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.558512 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.559120 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.560027 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.560242 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.560327 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.562980 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.562981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.562982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.562999 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.564568 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.565069 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.565071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.565071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.565142 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.565397 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.565398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.565399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.565521 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.565598 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.565884 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.565973 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:01:57.566344 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.556717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.21.66.124:46137)
0620 23:01:57.557020 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.557031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.557067 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:01:57.557653 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.557660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.557682 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.557805 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.558490 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.559265 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.559970 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.560273 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.560400 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.563288 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.563296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.563296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.563303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.564999 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.565356 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.565359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.565360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.565418 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.565621 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.565622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.565623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.565772 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.565923 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.566005 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.566086 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":89,"threads_started":1}
I20260620 23:01:57.566344 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.557194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41248)
0620 23:01:57.557435 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.557480 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.557655 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.557853 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.557856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.558215 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.558413 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.558418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.559228 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.559229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.559931 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.559933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.560356 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.560358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.560423 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.561067 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.561076 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.562876 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.564669 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.564675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.564677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.564991 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.565210 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.565211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.565211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.565329 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.565588 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.565594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.565741 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.565848 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.566008 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260620 23:01:57.566984 28605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.124@KRBTEST.COM for oryx/127.21.66.125@KRBTEST.COM
I20260620 23:01:57.574219 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.567230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.0.0.1:35042)
0620 23:01:57.567326 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.567336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.567377 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.567480 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.567481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.567594 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.567635 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.567636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.568495 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.568496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.569040 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.569042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.569466 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.569467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.569536 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.569971 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.569977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.571292 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.573034 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.573035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.573035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.573220 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.573460 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.573461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.573462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.573527 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.573813 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.573814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.573868 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.573954 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.574063 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:01:57.574219 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.567286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.21.66.125:44163)
0620 23:01:57.567344 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.567345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.567366 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:01:57.567421 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.567422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.567432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.567480 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.567722 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.568284 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.569148 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.569374 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.569467 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.571398 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.571400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.571401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.571409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.572869 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.573313 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.573314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.573315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.573387 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.573613 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.573614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.573614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.573742 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.573823 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.573993 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.574082 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:01:57.574852 28605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.124@KRBTEST.COM for oryx/127.21.66.124@KRBTEST.COM
I20260620 23:01:57.582335 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.575137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.0.0.1:52974)
0620 23:01:57.575197 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.575205 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.575244 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.575393 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.575394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.575533 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.575584 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.575586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.576276 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.576277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.576794 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.576796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.577408 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.577409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.577513 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.577932 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.577936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.579537 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.581320 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.581320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.581321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.581552 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.581736 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.581737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.581737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.581788 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.581997 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.581998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.582037 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.582169 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.582265 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:01:57.582525 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.575192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52974 (local address 127.21.66.124:46137)
0620 23:01:57.575230 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.575232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.575242 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:57.575301 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.575302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.575312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.575386 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.575660 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.576199 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.576884 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.577198 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.577266 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.579614 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.579616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.579616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.579622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.581224 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.581625 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.581626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.581626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.581679 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.581827 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.581828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.581828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.581925 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.581972 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.582202 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.582316 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":20}
I20260620 23:01:57.583938 28605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Bootstrap starting.
I20260620 23:01:57.584797 28605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:57.585099 28605 log.cc:824] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:57.585961 28605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: No bootstrap required, opened a new log
I20260620 23:01:57.587468 28605 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } }
I20260620 23:01:57.587610 28605 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:57.587633 28605 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd59e730385467685ff491c0ff20e3c, State: Initialized, Role: FOLLOWER
I20260620 23:01:57.587817 28605 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } }
I20260620 23:01:57.588099 28610 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } } }
I20260620 23:01:57.588156 28605 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:57.588183 28610 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:01:57.590005 28621 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:01:57.590072 28621 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:01:57.592800 28537 master_runner.cc:428] Detected that this master cfd59e730385467685ff491c0ff20e3c is joining an existing cluster
I20260620 23:01:57.592871 28537 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.66.124:46137
I20260620 23:01:57.618068 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27947
I20260620 23:01:57.623860 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:41777
--local_ip_for_outbound_sockets=127.21.66.65
--tserver_master_addrs=127.21.66.126:34795
--webserver_port=33673
--webserver_interface=127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab
--principal=oryx/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:01:57.630102 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.594298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.21.66.124:46137)
0620 23:01:57.594623 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.594625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.594633 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:57.594651 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.594652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.594660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.594716 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.595129 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.595862 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.596923 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.597232 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.597308 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.599214 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.599215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.599215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.599221 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.628238 (+ 29017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.629028 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.629029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.629029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.629074 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.629240 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.629241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.629241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.629366 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.629420 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.629919 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.630000 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260620 23:01:57.631632 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.594148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.21.66.126:34795)
0620 23:01:57.594204 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.594206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.594983 (+ 777us) server_negotiation.cc:407] Connection header received
0620 23:01:57.595097 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.595098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.595111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.595144 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.595714 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.596552 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.598546 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.598762 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.598854 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.626997 (+ 28143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.627000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.627000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.627009 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.629161 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.629683 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.629685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.629685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.629749 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.630316 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.630318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.630318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.631047 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.631126 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.631404 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.631500 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:01:57.631696 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.594281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41250)
0620 23:01:57.594918 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.594929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.595012 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.595216 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.595217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.595544 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.595612 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.595614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.596660 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.596661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.598454 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.598456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.598855 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.598856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.598933 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.599421 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.599426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.626882 (+ 27456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.629308 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.629309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.629310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.629568 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.630090 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.630091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.630092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.630211 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.631159 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.631160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.631227 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.631288 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.631591 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":1067,"thread_start_us":79,"threads_started":1}
I20260620 23:01:57.632072 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.594046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.0.0.1:52984)
0620 23:01:57.594309 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.594320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.594387 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.594757 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.594758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.594971 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.595040 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.595042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.595956 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.595957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.596840 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.596842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.597330 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.597332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.597406 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.598676 (+ 1270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.598681 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.599119 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.628339 (+ 29220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.628340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.628341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.628954 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.629135 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.629135 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.629136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.629181 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.629459 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.629460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.629494 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.630625 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.631975 (+ 1350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":132,"threads_started":1}
I20260620 23:01:57.633525 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.594169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.21.66.125:44163)
0620 23:01:57.594224 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.594226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.594983 (+ 757us) server_negotiation.cc:407] Connection header received
0620 23:01:57.595096 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.595097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.595109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.595144 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.595714 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.596563 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.597733 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.598035 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.598110 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.629607 (+ 31497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.629609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.629610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.629619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.631326 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.632663 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.632664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.632665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.632728 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.632975 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.632976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.632976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.633111 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.633151 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.633344 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.633436 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:01:57.633834 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.594770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.0.0.1:35044)
0620 23:01:57.594915 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.594927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.595012 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.595220 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.595220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.595538 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.595613 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.595614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.596660 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.596661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.597647 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.597650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.598154 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.598155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.598227 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.626905 (+ 28678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.626912 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.628706 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.631430 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.631431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.631432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.631916 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.632800 (+ 884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.632801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.632801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.632873 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.633196 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.633197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.633239 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.633633 (+ 394us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.633733 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":88,"thread_start_us":92,"threads_started":1}
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996516, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.126@KRBTEST.COM for oryx/127.21.66.124@KRBTEST.COM
I20260620 23:01:57.642724 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.634747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.21.66.124:46137)
0620 23:01:57.634816 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.634818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.634826 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:57.634859 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.634860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.634879 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.634934 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.635302 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.636032 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.637113 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.637429 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.637502 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.639808 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.639810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.639810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.639818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.641451 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.641863 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.641864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.641865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.641930 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.642163 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.642164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.642165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.642300 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.642366 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.642550 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.642628 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:01:57.642729 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.634667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.0.0.1:52998)
0620 23:01:57.634710 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.634722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.634795 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.634969 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.634970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.635147 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.635195 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.635197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.636126 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.636128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.637032 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.637035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.637526 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.637527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.637598 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.638050 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.638055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.639714 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.641547 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.641548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.641549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.641783 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.642002 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.642003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.642004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.642074 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.642373 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.642374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.642439 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.642515 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.642623 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 23:01:57.644073 27885 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.66.124:46137: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cfd59e730385467685ff491c0ff20e3c"
member_type: NON_VOTER
last_known_addr {
host: "127.21.66.124"
port: 46137
}
attrs {
promote: true
}
}
dest_uuid: "05367f8fccc5489692b9e4debe1b467f"
cas_config_opid_index: 5
I20260620 23:01:57.644315 27885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } }
I20260620 23:01:57.644666 28444 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cfd59e730385467685ff491c0ff20e3c (127.21.66.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.645102 28579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:57.645602 28444 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.66.124:46137
I20260620 23:01:57.645663 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.645723 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:57.646049 28579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Refusing update from remote peer 05367f8fccc5489692b9e4debe1b467f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:01:57.646288 28537 master.cc:537] Master@127.21.66.124:46137 shutting down...
I20260620 23:01:57.646410 28445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:01:57.647086 27869 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f -> Peer cfd59e730385467685ff491c0ff20e3c (127.21.66.124:46137): Couldn't send request to peer cfd59e730385467685ff491c0ff20e3c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 23:01:57.649768 28537 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:57.649863 28537 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:57.649904 28537 tablet_replica.cc:330] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: stopping tablet replica
I20260620 23:01:57.651577 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.644734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.0.0.1:35050)
0620 23:01:57.647196 (+ 2462us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.647206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.647248 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.647476 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.647477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.647479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:57.647521 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.647523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.648506 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.648507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.649906 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.649909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.650936 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.650937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.651161 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.651189 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.651454 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.651503 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2449}
I20260620 23:01:57.651832 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.644819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.21.66.125:44163)
0620 23:01:57.647167 (+ 2348us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.647169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.647297 (+ 128us) server_negotiation.cc:407] Connection header received
0620 23:01:57.647322 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.647323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.647347 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.647415 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:57.647590 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.648356 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.650016 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.650643 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.650690 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.650757 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.651722 (+ 965us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.651769 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2320}
I20260620 23:01:57.652113 28493 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 LEARNER]: Refusing update from remote peer 05367f8fccc5489692b9e4debe1b467f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:01:57.652338 28445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 23:01:57.652921 28493 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cfd59e730385467685ff491c0ff20e3c (127.21.66.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.653671 28537 master.cc:552] Master@127.21.66.124:46137 shutdown complete.
I20260620 23:01:57.653705 28537 master_runner.cc:305] Clearing existing system tablet
I20260620 23:01:57.653829 28537 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.653865 28537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.654083 28522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05367f8fccc5489692b9e4debe1b467f. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.654209 28522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:01:57.654245 28522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: NON_VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: true } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.654286 28522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:01:57.654748 28537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:01:57.654829 28445 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: attempting to promote NON_VOTER 7ca6226bb5854c73a510b88457ae8789 to VOTER
I20260620 23:01:57.655050 28445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } }
I20260620 23:01:57.655489 28634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.655622 28537 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.655661 28537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:01:57.655664 28493 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 LEARNER]: Refusing update from remote peer 05367f8fccc5489692b9e4debe1b467f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:01:57.655681 28537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
uuid: "cfd59e730385467685ff491c0ff20e3c"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4c3bd9a1dfebadfa2f526a7e86cd51b9"
server_key_iv: "eeab5969f4155feab8218fdc652dd311"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.655910 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.655515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.0.0.1:53006)
0620 23:01:57.655826 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.655881 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:46137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260620 23:01:57.656168 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.656102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.0.0.1:53018)
0620 23:01:57.656130 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.656149 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:46137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 23:01:57.656216 27869 consensus_peers.cc:487] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f -> Peer cfd59e730385467685ff491c0ff20e3c (127.21.66.124:46137): Couldn't send request to peer cfd59e730385467685ff491c0ff20e3c. Status: Network error: Client connection negotiation failed: client connection to 127.21.66.124:46137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:01:57.656339 28537 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:57.658066 28537 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:01:57.658159 28537 log.cc:1196] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 23:01:57.658341 28537 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Deleting consensus metadata
I20260620 23:01:57.658445 28537 master_runner.cc:315] Copying system tablet from 127.21.66.126:34795
I20260620 23:01:57.659049 28537 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: tablet copy: Beginning tablet copy session from remote peer at address 127.21.66.126:34795
I20260620 23:01:57.659283 28635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:01:57.660064 28444 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ca6226bb5854c73a510b88457ae8789 (127.21.66.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.660223 28493 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7ca6226bb5854c73a510b88457ae8789 (127.21.66.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.661023 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.661100 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:57.661145 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.661113 28632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.661181 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:57.661197 28632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:57.676038 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.659689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.21.66.126:34795)
0620 23:01:57.659726 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.659728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.668816 (+ 9088us) server_negotiation.cc:407] Connection header received
0620 23:01:57.668864 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.668865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.668886 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.668964 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.669352 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.670162 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.671212 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.671481 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.671560 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.672693 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.672694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.672695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.672703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.674507 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.674983 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.674985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.674985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.675054 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.675304 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.675306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.675306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.675510 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.675581 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.675847 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.675940 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:57.676062 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.659769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41254)
0620 23:01:57.668691 (+ 8922us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.668711 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.668809 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.668987 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.668987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.669208 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.669267 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.669269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.670268 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.670269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.671111 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.671114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.671573 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.671574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.671635 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.672183 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.672188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.672590 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.674604 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.674605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.674606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.674885 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.675138 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.675139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.675139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.675222 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.675611 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.675613 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.675683 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.675811 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.675932 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8882,"thread_start_us":40,"threads_started":1}
I20260620 23:01:57.676635 27905 tablet_copy_service.cc:140] P 05367f8fccc5489692b9e4debe1b467f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cfd59e730385467685ff491c0ff20e3c ({username='oryx', principal='oryx/127.21.66.124@KRBTEST.COM'} at 127.0.0.1:41254)
I20260620 23:01:57.676714 27905 tablet_copy_service.cc:161] P 05367f8fccc5489692b9e4debe1b467f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cfd59e730385467685ff491c0ff20e3c at {username='oryx', principal='oryx/127.21.66.124@KRBTEST.COM'} at 127.0.0.1:41254: session id = cfd59e730385467685ff491c0ff20e3c-00000000000000000000000000000000
I20260620 23:01:57.677285 27905 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:01:57.677850 28537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:01:57.679293 28537 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: tablet copy: Starting download of 0 data blocks...
I20260620 23:01:57.679448 28537 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: tablet copy: Starting download of 1 WAL segments...
I20260620 23:01:57.680227 28537 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:01:57.681232 27905 tablet_copy_service.cc:342] P 05367f8fccc5489692b9e4debe1b467f: Request end of tablet copy session cfd59e730385467685ff491c0ff20e3c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.66.124@KRBTEST.COM'} at 127.0.0.1:41254
I20260620 23:01:57.681303 27905 tablet_copy_service.cc:433] P 05367f8fccc5489692b9e4debe1b467f: ending tablet copy session cfd59e730385467685ff491c0ff20e3c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cfd59e730385467685ff491c0ff20e3c
I20260620 23:01:57.683614 28537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:57.684873 28644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.684942 28643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.685171 28646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.685366 28537 server_base.cc:1034] running on GCE node
I20260620 23:01:57.685495 28537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:57.685670 28537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:57.686841 28537 hybrid_clock.cc:648] HybridClock initialized: now 1781996517686829 us; error 25 us; skew 500 ppm
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:57.688489 28537 init.cc:377] Logged in from keytab as oryx/127.21.66.124@KRBTEST.COM (short username oryx)
I20260620 23:01:57.688771 28537 webserver.cc:460] Webserver started at http://127.21.66.124:41567/ using document root <none> and password file <none>
I20260620 23:01:57.688908 28537 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.688967 28537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.689755 28537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:57.690405 28653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.690577 28537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.690634 28537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
uuid: "cfd59e730385467685ff491c0ff20e3c"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "4c3bd9a1dfebadfa2f526a7e86cd51b9"
server_key_iv: "eeab5969f4155feab8218fdc652dd311"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.690699 28537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.699720 28537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:57.701606 28537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:57.704720 28537 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.124:46137
I20260620 23:01:57.705137 28537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
I20260620 23:01:57.706069 28706 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:01:57.706670 28706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Bootstrap starting.
I20260620 23:01:57.708039 28705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.124:46137 every 8 connection(s)
I20260620 23:01:57.709934 28706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:01:57.710227 28706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: Bootstrap complete.
I20260620 23:01:57.710462 28706 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } }
I20260620 23:01:57.710561 28706 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfd59e730385467685ff491c0ff20e3c, State: Initialized, Role: LEARNER
I20260620 23:01:57.710599 28706 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } }
I20260620 23:01:57.710808 28706 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:01:57.712445 28720 catalog_manager.cc:1259] Loaded cluster ID: daaf26df389f444f9e25258b158619b8
I20260620 23:01:57.712563 28720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: loading cluster ID for follower catalog manager: success
I20260620 23:01:57.712949 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:57.713406 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: This master's current role is: LEARNER
I20260620 23:01:57.714675 28720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: acquiring CA information for follower catalog manager: success
I20260620 23:01:57.715049 28720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:01:57.731525 28630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:57.731787 28630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:57.731830 28630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:57.733042 28630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:57.733111 28630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:57.733160 28630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:01:57.734779 28630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab
--principal=oryx/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:41777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=33673
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:57.735073 28630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:57.735320 28630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:57.741564 28726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.741647 28630 server_base.cc:1034] running on GCE node
W20260620 23:01:57.741626 28728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.741791 28725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.742048 28630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:57.742414 28630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:57.743634 28630 hybrid_clock.cc:648] HybridClock initialized: now 1781996517743609 us; error 37 us; skew 500 ppm
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:57.746822 28630 init.cc:377] Logged in from keytab as oryx/127.21.66.65@KRBTEST.COM (short username oryx)
I20260620 23:01:57.747474 28630 webserver.cc:460] Webserver started at http://127.21.66.65:33673/ using document root <none> and password file <none>
I20260620 23:01:57.747745 28630 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.747820 28630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.749509 28630 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:57.750183 28735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.750376 28630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:57.750468 28630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "bc8d4962f326499fb2dd201cc3c1ef88"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "cb6142e568a6558cb8cf6d3e194baec7"
server_key_iv: "280978210fc9341e6ab32eb2f5d422e6"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.750892 28630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.768118 28630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:57.770764 28630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:57.770886 28630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:57.771092 28630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:57.771430 28630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:57.771469 28630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.771492 28630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:57.771504 28630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.777555 28630 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:41777
I20260620 23:01:57.777653 28848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:41777 every 8 connection(s)
I20260620 23:01:57.777990 28630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:01:57.780470 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 28630
I20260620 23:01:57.780648 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 28081
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.125@KRBTEST.COM
I20260620 23:01:57.787057 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:36603
--local_ip_for_outbound_sockets=127.21.66.66
--tserver_master_addrs=127.21.66.126:34795
--webserver_port=40493
--webserver_interface=127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab
--principal=oryx/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.124@KRBTEST.COM
I20260620 23:01:57.792034 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.778874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:58829 (local address 127.21.66.125:44163)
0620 23:01:57.778925 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.778927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.779226 (+ 299us) server_negotiation.cc:407] Connection header received
0620 23:01:57.780451 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.780453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.780464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.780514 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.781365 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.782233 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.783841 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.784207 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.784250 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.788266 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.788267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.788268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.788278 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.790221 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.790746 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.790748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.790748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.790811 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.791130 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.791131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.791131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.791359 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.791406 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.791824 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.791911 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:01:57.792052 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.778996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:60251 (local address 127.21.66.126:34795)
0620 23:01:57.779040 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.779042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.779175 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:01:57.780291 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.780293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.780307 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.780385 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.782636 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.783393 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.784363 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.784654 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.784698 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.788268 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.788270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.788270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.788279 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.790093 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.790919 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.790920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.790920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.790977 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.791249 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.791250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.791251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.791388 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.791434 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.791822 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.791930 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:01:57.792208 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.779042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:45165 (local address 127.21.66.124:46137)
0620 23:01:57.779178 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.779179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.780494 (+ 1315us) server_negotiation.cc:407] Connection header received
0620 23:01:57.780603 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.780604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.780620 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.781339 (+ 719us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.781382 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.782507 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.783823 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.784150 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.784210 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.788267 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.788268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.788269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.788278 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.790934 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.791320 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.791321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.791321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.791384 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.791646 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.791647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.791648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.791792 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.791835 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.792022 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.792108 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":54,"threads_started":1}
I20260620 23:01:57.792753 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.778762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:60251)
0620 23:01:57.779035 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.779081 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.780156 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.780482 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.780486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.782468 (+ 1982us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.782536 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.782538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.783493 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.783494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.784272 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.784274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.784775 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.784776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.784839 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.785249 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.785254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.788142 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.790245 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.790252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.790254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.790665 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.791054 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.791055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.791056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.791125 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.791518 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.791519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.791658 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.792161 (+ 503us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.792506 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":162,"thread_start_us":107,"threads_started":1}
I20260620 23:01:57.792753 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.778762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.21.66.65:45165)
0620 23:01:57.780378 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.780389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.780446 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.780742 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.780742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.780978 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.781256 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.781263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.782604 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.782605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.783737 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.783739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.784279 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.784281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.784435 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.785383 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.785387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.788174 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.791005 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.791006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.791006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.791222 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.791458 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.791458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.791459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.791513 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.791881 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.791882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.791919 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.792343 (+ 424us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.792440 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1548,"mutex_wait_us":2,"spinlock_wait_cycles":8960,"thread_start_us":123,"threads_started":1}
I20260620 23:01:57.792753 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.778762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.21.66.65:58829)
0620 23:01:57.779175 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.779184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.780154 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.780566 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.780567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.781186 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.781269 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.781272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.782363 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.782364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.783741 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.783744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.784342 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.784343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.784438 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.785143 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.785153 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.788155 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.790296 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.790297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.790297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.790665 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.790883 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.790883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.790883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.791012 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.791475 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.791482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.791658 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.792172 (+ 514us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.792399 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":4,"thread_start_us":89,"threads_started":1}
I20260620 23:01:57.793284 28850 heartbeater.cc:347] Connected to a master server at 127.21.66.124:46137
I20260620 23:01:57.793383 28850 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:57.793594 28850 heartbeater.cc:511] Master 127.21.66.124:46137 requested a full tablet report, sending...
I20260620 23:01:57.793696 28849 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34795
I20260620 23:01:57.793749 28849 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:57.793818 28849 heartbeater.cc:511] Master 127.21.66.126:34795 requested a full tablet report, sending...
I20260620 23:01:57.793853 28851 heartbeater.cc:347] Connected to a master server at 127.21.66.125:44163
I20260620 23:01:57.794077 27885 ts_manager.cc:194] Re-registered known tserver with Master: bc8d4962f326499fb2dd201cc3c1ef88 (127.21.66.65:41777)
I20260620 23:01:57.794108 28670 ts_manager.cc:194] Registered new tserver with Master: bc8d4962f326499fb2dd201cc3c1ef88 (127.21.66.65:41777)
I20260620 23:01:57.794265 28851 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:57.794337 28851 heartbeater.cc:511] Master 127.21.66.125:44163 requested a full tablet report, sending...
I20260620 23:01:57.794624 27885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:60251
I20260620 23:01:57.794843 28483 ts_manager.cc:194] Registered new tserver with Master: bc8d4962f326499fb2dd201cc3c1ef88 (127.21.66.65:41777)
W20260620 23:01:57.877274 28857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:57.877494 28857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:57.877521 28857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:57.878701 28857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:57.878739 28857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:57.878765 28857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:01:57.880195 28857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab
--principal=oryx/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:36603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=40493
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:57.880395 28857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:57.880590 28857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:57.885833 28862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:57.885840 28863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.885984 28857 server_base.cc:1034] running on GCE node
W20260620 23:01:57.885921 28865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:57.886377 28857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:57.886675 28857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:57.887912 28857 hybrid_clock.cc:648] HybridClock initialized: now 1781996517887882 us; error 55 us; skew 500 ppm
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:57.890906 28857 init.cc:377] Logged in from keytab as oryx/127.21.66.66@KRBTEST.COM (short username oryx)
I20260620 23:01:57.891456 28857 webserver.cc:460] Webserver started at http://127.21.66.66:40493/ using document root <none> and password file <none>
I20260620 23:01:57.891855 28857 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:57.891954 28857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:57.893563 28857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 23:01:57.894606 28872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.894923 28857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:57.895068 28857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "ce78aa9ed2f745968b8f29b347691907"
format_stamp: "Formatted at 2026-06-20 23:01:56 on dist-test-slave-qgtk"
server_key: "03ed749f0fb139f2c4fd22aad1910a4d"
server_key_iv: "c00a564a294150ab03d524579f490501"
server_key_version: "encryptionkey@0"
I20260620 23:01:57.895500 28857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:57.918315 28857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:57.920964 28857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:57.921149 28857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:57.921411 28857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:57.921751 28857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:57.921805 28857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.921852 28857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:57.921882 28857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:57.928350 28857 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:36603
I20260620 23:01:57.928406 28985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:36603 every 8 connection(s)
I20260620 23:01:57.928824 28857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.124@KRBTEST.COM
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
Jun 20 23:01:57 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996517, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.125@KRBTEST.COM
I20260620 23:01:57.938088 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 28857
I20260620 23:01:57.938282 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 28215
I20260620 23:01:57.941309 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.929646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:47077 (local address 127.21.66.124:46137)
0620 23:01:57.929707 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.929710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.929938 (+ 228us) server_negotiation.cc:407] Connection header received
0620 23:01:57.931086 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.931088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.931105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.931155 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.932059 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.932569 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.933948 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.934312 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.934378 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.937022 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.937023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.937024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.937031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.939560 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940098 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.940100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.940100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.940162 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940467 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.940468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.940468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.940635 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.940681 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.941103 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.941209 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:01:57.941421 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.929659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:60155 (local address 127.21.66.126:34795)
0620 23:01:57.929752 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.929754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.929918 (+ 164us) server_negotiation.cc:407] Connection header received
0620 23:01:57.931222 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.931224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.931235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.931290 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.932073 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.932849 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.934010 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.934337 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.934404 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.937579 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.937580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.937581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.937589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.940222 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940635 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.940637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.940637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.940694 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940887 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.940888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.940888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.941023 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.941067 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.941261 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.941334 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:01:57.941465 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.930132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:37463 (local address 127.21.66.125:44163)
0620 23:01:57.930183 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:57.930185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:57.930195 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:57.931347 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:57.931348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:57.931360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:57.931428 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:57.932121 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.932845 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.933837 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.934118 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.934236 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.938015 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:57.938017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:57.938017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:57.938026 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:57.940108 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940587 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.940589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.940589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.940648 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940837 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:57.940838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:57.940839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:57.940977 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:57.941022 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:57.941236 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 23:01:57.941322 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:01:57.941843 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.929611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.21.66.66:47077)
0620 23:01:57.929788 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.929833 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.930960 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.931286 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.931290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.931742 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.931956 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.931961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.932681 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.932682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.933862 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.933865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.934389 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.934390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.934463 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.935129 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.935134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.936874 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.939661 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.939670 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.939673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.939999 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.940238 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.940239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.940385 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.940731 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.940737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.940932 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.941401 (+ 469us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.941619 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260620 23:01:57.942023 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.929605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.66:60155)
0620 23:01:57.929807 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.929832 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.930954 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.931309 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.931310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.931835 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.931961 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.931963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.933161 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.933162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.933948 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.933950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.934459 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.934459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.934537 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.935240 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.935244 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.937401 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.940302 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.940303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.940554 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.940766 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.940766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.940821 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.941115 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.941116 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.941149 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.941529 (+ 380us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.941916 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":333,"thread_start_us":105,"threads_started":1}
I20260620 23:01:57.942458 28986 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34795
I20260620 23:01:57.942468 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:57.929939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.21.66.66:37463)
0620 23:01:57.930069 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:57.930085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:57.930961 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:57.931432 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:57.931433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:57.931839 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:57.931961 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.931963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.932925 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.932926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.933738 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:57.933741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:57.934241 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:57.934242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:57.934380 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:57.935095 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:57.935103 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:57.937902 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:57.940198 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.940200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.940498 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.940717 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:57.940718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:57.940718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:57.940776 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:57.941068 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:57.941069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:57.941113 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:01:57.942051 (+ 938us) client_negotiation.cc:240] Negotiation successful
0620 23:01:57.942311 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":6,"thread_start_us":76,"threads_started":1}
I20260620 23:01:57.942607 28986 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:57.942922 28986 heartbeater.cc:511] Master 127.21.66.126:34795 requested a full tablet report, sending...
I20260620 23:01:57.943063 28988 heartbeater.cc:347] Connected to a master server at 127.21.66.125:44163
I20260620 23:01:57.943125 28988 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:57.943199 28988 heartbeater.cc:511] Master 127.21.66.125:44163 requested a full tablet report, sending...
I20260620 23:01:57.943307 27885 ts_manager.cc:194] Re-registered known tserver with Master: ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:01:57.943500 28483 ts_manager.cc:194] Registered new tserver with Master: ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:01:57.943786 27885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:60155
I20260620 23:01:57.944787 28987 heartbeater.cc:347] Connected to a master server at 127.21.66.124:46137
I20260620 23:01:57.944844 28987 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:57.944911 28987 heartbeater.cc:511] Master 127.21.66.124:46137 requested a full tablet report, sending...
I20260620 23:01:57.945235 28670 ts_manager.cc:194] Registered new tserver with Master: ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:01:57.946239 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:38565
--local_ip_for_outbound_sockets=127.21.66.67
--tserver_master_addrs=127.21.66.126:34795
--webserver_port=45045
--webserver_interface=127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab
--principal=oryx/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:01:58.032308 28993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:01:58.032545 28993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:01:58.032572 28993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:01:58.033792 28993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:01:58.033839 28993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:01:58.033867 28993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:01:58.035485 28993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab
--principal=oryx/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:38565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=45045
--tserver_master_addrs=127.21.66.126:34795,127.21.66.125:44163,127.21.66.124:46137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:01:58.035742 28993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:01:58.036031 28993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:01:58.041702 28998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:58.041708 29001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:01:58.041702 28999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:01:58.041759 28993 server_base.cc:1034] running on GCE node
I20260620 23:01:58.042192 28993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:01:58.042454 28993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:01:58.043654 28993 hybrid_clock.cc:648] HybridClock initialized: now 1781996518043634 us; error 27 us; skew 500 ppm
Jun 20 23:01:58 dist-test-slave-qgtk krb5kdc[27840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996518, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:01:58.046965 28993 init.cc:377] Logged in from keytab as oryx/127.21.66.67@KRBTEST.COM (short username oryx)
I20260620 23:01:58.047598 28993 webserver.cc:460] Webserver started at http://127.21.66.67:45045/ using document root <none> and password file <none>
I20260620 23:01:58.047699 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.044559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.0.0.1:53034)
0620 23:01:58.044659 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.044670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.044721 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.044928 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.044929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.044931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:58.045034 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.045037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.045670 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.045671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.046684 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.046689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.047388 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.047389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.047460 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.047465 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.047566 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.047619 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:01:58.047745 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.044642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.21.66.124:46137)
0620 23:01:58.044746 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.044748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.044792 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:01:58.044828 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.044829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.044847 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.044899 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:58.045112 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.045586 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.046781 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.047305 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.047364 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.047418 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.047600 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.047673 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:01:58.047885 28993 fs_manager.cc:359] Metadata directory not provided
I20260620 23:01:58.048020 28993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:01:58.048051 28680 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 23:01:58.048246 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05367f8fccc5489692b9e4debe1b467f. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:58.048383 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: This master's current role is: LEARNER
I20260620 23:01:58.049693 28993 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:01:58.050458 29008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:58.050673 28993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:58.050781 28993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "d2bbd190c5d24aeea1c0eb118d633f6a"
format_stamp: "Formatted at 2026-06-20 23:01:57 on dist-test-slave-qgtk"
server_key: "df5bc0554fc845414f6ec5f53f2d5233"
server_key_iv: "3369bd8872b80382460e3c5e3c5b1dae"
server_key_version: "encryptionkey@0"
I20260620 23:01:58.051155 28993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:01:58.061143 28993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:01:58.063517 28993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:01:58.063738 28993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:01:58.063997 28993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:01:58.064311 28993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:01:58.064373 28993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:58.064437 28993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:01:58.064468 28993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:01:58.071069 28993 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:38565
I20260620 23:01:58.071110 29121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:38565 every 8 connection(s)
I20260620 23:01:58.071470 28993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:01:58 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996518, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
Jun 20 23:01:58 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996518, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.125@KRBTEST.COM
Jun 20 23:01:58 dist-test-slave-qgtk krb5kdc[27840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996518, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.124@KRBTEST.COM
I20260620 23:01:58.082194 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 28993
I20260620 23:01:58.083544 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.072604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:36995 (local address 127.21.66.126:34795)
0620 23:01:58.072687 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.072690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.072702 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:01:58.073372 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.073373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.073393 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.073433 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.074840 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.075620 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.076635 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.076986 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.077050 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.079640 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.079642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.079643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.079653 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.081589 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082221 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.082222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.082223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.082346 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082684 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.082685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.082685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.082847 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.082911 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.083355 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.083451 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:01:58.083586 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.072377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:59991 (local address 127.21.66.125:44163)
0620 23:01:58.072426 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.072428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.072864 (+ 436us) server_negotiation.cc:407] Connection header received
0620 23:01:58.073303 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.073304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.073318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.073368 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.074235 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.074753 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.076264 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.076556 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.076615 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.080019 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.080020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.080020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.080028 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.081878 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082605 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.082606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.082607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.082678 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082922 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.082923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.082923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.083075 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.083139 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.083371 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.083465 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:01:58.084079 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.072314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:44741 (local address 127.21.66.124:46137)
0620 23:01:58.072373 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.072376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.072724 (+ 348us) server_negotiation.cc:407] Connection header received
0620 23:01:58.073549 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.073551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.073564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.073627 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.074306 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.074819 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.076182 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.076498 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.076563 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.080787 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.080789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.080789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.080798 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.082760 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.083194 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.083195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.083196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.083260 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.083480 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.083481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.083481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.083615 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.083664 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.083884 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.083969 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:01:58.084187 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.072276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.21.66.67:59991)
0620 23:01:58.072785 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.072794 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.073195 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.073894 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.073895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.074055 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:58.074131 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.074138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.074865 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.074866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.076123 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.076126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.076661 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.076662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.076792 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.077454 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:58.077465 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:58.079915 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:58.082208 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:58.082210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:58.082474 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:58.082764 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:58.082765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:58.082837 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:58.083178 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:58.083178 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:58.083235 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.083699 (+ 464us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.083918 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":323,"thread_start_us":76,"threads_started":1}
I20260620 23:01:58.084406 29126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.072276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.21.66.67:44741)
0620 23:01:58.072492 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.072546 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.073179 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.073662 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.073663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.073898 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:58.074111 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.074161 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.074914 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.074915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.076101 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.076103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.076631 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.076632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.076811 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.077868 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:58.077873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:58.080639 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:58.082854 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082854 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:58.082855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:58.083084 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:58.083341 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:58.083342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:58.083342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:58.083400 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:58.083720 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:58.083720 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:58.083763 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.084165 (+ 402us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.084283 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260620 23:01:58.084901 29124 heartbeater.cc:347] Connected to a master server at 127.21.66.125:44163
I20260620 23:01:58.084991 29123 heartbeater.cc:347] Connected to a master server at 127.21.66.124:46137
I20260620 23:01:58.085013 29124 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:58.085026 29123 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:58.085269 29124 heartbeater.cc:511] Master 127.21.66.125:44163 requested a full tablet report, sending...
I20260620 23:01:58.085270 29123 heartbeater.cc:511] Master 127.21.66.124:46137 requested a full tablet report, sending...
I20260620 23:01:58.085714 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.072279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.67:36995)
0620 23:01:58.072629 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.072638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.073179 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.073459 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.073464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.073939 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:01:58.074151 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.074153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.075694 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.075695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.076539 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.076541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.077069 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.077070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.077156 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.077597 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:01:58.077625 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:01:58.079434 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:01:58.081710 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:58.081721 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:58.081724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:58.082092 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:58.082434 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:01:58.082435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:01:58.082435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:01:58.082605 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:01:58.082949 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:01:58.082958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:01:58.083166 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.084086 (+ 920us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.085260 (+ 1174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":90,"threads_started":1}
I20260620 23:01:58.085745 28483 ts_manager.cc:194] Registered new tserver with Master: d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565)
I20260620 23:01:58.085745 28670 ts_manager.cc:194] Registered new tserver with Master: d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565)
I20260620 23:01:58.086338 29122 heartbeater.cc:347] Connected to a master server at 127.21.66.126:34795
I20260620 23:01:58.086402 29122 heartbeater.cc:464] Registering TS with master...
I20260620 23:01:58.086488 29122 heartbeater.cc:511] Master 127.21.66.126:34795 requested a full tablet report, sending...
I20260620 23:01:58.086745 27885 ts_manager.cc:194] Re-registered known tserver with Master: d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565)
I20260620 23:01:58.087307 27885 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:36995
I20260620 23:01:58.092509 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.083696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.21.66.125:44163)
0620 23:01:58.083745 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.083747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.084736 (+ 989us) server_negotiation.cc:407] Connection header received
0620 23:01:58.084795 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.084795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.084806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.084860 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.085726 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.086504 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.087626 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.087918 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.087958 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.089543 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.089544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.089545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.089552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.091340 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.091729 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.091730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.091731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.091788 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.091996 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.091997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.091997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.092115 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.092223 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.092326 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.092404 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:01:58.094717 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.084098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.21.66.126:34795)
0620 23:01:58.084156 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.084158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.086257 (+ 2099us) server_negotiation.cc:407] Connection header received
0620 23:01:58.086359 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.086361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.086374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.086434 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.087245 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.088221 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.089272 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.089605 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.089652 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.091481 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.091482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.091482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.091490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.093403 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.093811 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.093812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.093812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.093869 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.094063 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.094064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.094064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.094272 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.094390 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.094509 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.094607 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:01:58.095192 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.084474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.21.66.124:46137)
0620 23:01:58.085151 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.085153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.086511 (+ 1358us) server_negotiation.cc:407] Connection header received
0620 23:01:58.086645 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.086646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.086656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.086684 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.087485 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.088201 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.089305 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.089619 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.089691 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.091337 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.091339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.091339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.091348 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.093609 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.094001 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.094002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.094002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.094178 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.094397 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.094398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.094398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.094558 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.094660 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.094993 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.095077 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652}
I20260620 23:01:58.096323 27885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:58.097918 27885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:58.098775 28445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: NON_VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: true } } }
I20260620 23:01:58.098934 28445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:58.100495 28445 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: attempting to promote NON_VOTER cfd59e730385467685ff491c0ff20e3c to VOTER
I20260620 23:01:58.100626 28445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } }
I20260620 23:01:58.100997 28493 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Refusing update from remote peer 05367f8fccc5489692b9e4debe1b467f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 23:01:58.101066 28680 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 LEARNER]: Refusing update from remote peer 05367f8fccc5489692b9e4debe1b467f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 23:01:58.101212 28444 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 23:01:58.101357 28642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 23:01:58.102005 28635 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cfd59e730385467685ff491c0ff20e3c (127.21.66.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.102182 28493 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cfd59e730385467685ff491c0ff20e3c (127.21.66.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.102178 28680 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, cfd59e730385467685ff491c0ff20e3c (127.21.66.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.102891 28635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.102970 28635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:58.103024 28635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.103082 28635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:58.103128 28635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.103181 28635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:58.103127 28632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.103217 28632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:58.103358 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05367f8fccc5489692b9e4debe1b467f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:58.103444 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:58.108420 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.104131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:36603 (local address 127.0.0.1:44562)
0620 23:01:58.104411 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.104420 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.104468 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.104890 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.104891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.104893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:58.104962 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.104964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.105937 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.105938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.107203 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.107206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.108079 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.108080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.108173 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.108179 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.108236 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.108292 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20260620 23:01:58.108573 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.104209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.21.66.65:41777)
0620 23:01:58.104387 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.104393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.104594 (+ 201us) server_negotiation.cc:407] Connection header received
0620 23:01:58.104707 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.104709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.104724 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.104775 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:58.105006 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.106160 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.107238 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.107988 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.108037 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.108119 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.108414 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.108459 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260620 23:01:58.108805 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.104131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:41777 (local address 127.0.0.1:46322)
0620 23:01:58.104543 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.104551 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.104618 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.104865 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.104866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.104868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:58.104922 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.104924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.106262 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.106263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.107086 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.107088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.108182 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.108183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.108292 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.108296 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.108670 (+ 374us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.108723 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":46,"thread_start_us":111,"threads_started":1}
I20260620 23:01:58.109041 28443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.104060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:38565 (local address 127.0.0.1:45306)
0620 23:01:58.104194 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.104206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.104251 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.104742 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.104743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.104745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:58.104806 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.104809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.106285 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.106285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.107517 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.107520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.108784 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.108785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.108854 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.108858 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.108910 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.108957 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":53}
I20260620 23:01:58.109237 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.104327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.21.66.67:38565)
0620 23:01:58.104497 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.104502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.104513 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:01:58.104575 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.104577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.104595 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.104681 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:58.104967 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.106197 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.107696 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.108487 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.108782 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.108937 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.109024 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.109160 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260620 23:01:58.109452 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.104409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.21.66.66:36603)
0620 23:01:58.104595 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.104599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.104609 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:01:58.104644 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.104711 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.104731 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.104822 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:58.105047 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.105821 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.107309 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.107976 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.108079 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.108702 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.108743 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.108794 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260620 23:01:58.109570 28783 tablet_service.cc:1467] Processing CreateTablet for tablet 22beeac08d694eaa9d037e7a73963eb8 (DEFAULT_TABLE table=test-table [id=2ae9c6ca16bb4bf192db54bce06f948f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:58.110001 28783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22beeac08d694eaa9d037e7a73963eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:58.109967 29056 tablet_service.cc:1467] Processing CreateTablet for tablet 22beeac08d694eaa9d037e7a73963eb8 (DEFAULT_TABLE table=test-table [id=2ae9c6ca16bb4bf192db54bce06f948f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:58.110283 28920 tablet_service.cc:1467] Processing CreateTablet for tablet 22beeac08d694eaa9d037e7a73963eb8 (DEFAULT_TABLE table=test-table [id=2ae9c6ca16bb4bf192db54bce06f948f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:58.110632 28920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22beeac08d694eaa9d037e7a73963eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:58.110699 29056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22beeac08d694eaa9d037e7a73963eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:58.113833 29148 tablet_bootstrap.cc:492] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Bootstrap starting.
I20260620 23:01:58.114326 29149 tablet_bootstrap.cc:492] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Bootstrap starting.
I20260620 23:01:58.114735 29150 tablet_bootstrap.cc:492] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Bootstrap starting.
I20260620 23:01:58.114770 29148 tablet_bootstrap.cc:654] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:58.115013 29149 tablet_bootstrap.cc:654] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:58.115197 29148 log.cc:824] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:58.115343 29149 log.cc:824] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:58.115403 29150 tablet_bootstrap.cc:654] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:58.115693 29150 log.cc:824] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:01:58.116184 29149 tablet_bootstrap.cc:492] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: No bootstrap required, opened a new log
I20260620 23:01:58.116261 29149 ts_tablet_manager.cc:1397] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:58.116411 29150 tablet_bootstrap.cc:492] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: No bootstrap required, opened a new log
I20260620 23:01:58.116477 29150 ts_tablet_manager.cc:1397] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:01:58.118676 29150 raft_consensus.cc:348] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.118877 29150 raft_consensus.cc:374] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:58.118912 29150 raft_consensus.cc:729] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bbd190c5d24aeea1c0eb118d633f6a, State: Initialized, Role: FOLLOWER
I20260620 23:01:58.119076 29150 consensus_queue.cc:260] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.119124 29149 raft_consensus.cc:348] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.119290 29149 raft_consensus.cc:374] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:58.119330 29149 raft_consensus.cc:729] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8d4962f326499fb2dd201cc3c1ef88, State: Initialized, Role: FOLLOWER
I20260620 23:01:58.119369 29150 ts_tablet_manager.cc:1428] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:01:58.119424 29122 heartbeater.cc:503] Master 127.21.66.126:34795 was elected leader, sending a full tablet report...
I20260620 23:01:58.119446 29149 consensus_queue.cc:260] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.119721 29149 ts_tablet_manager.cc:1428] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 23:01:58.120131 28849 heartbeater.cc:503] Master 127.21.66.126:34795 was elected leader, sending a full tablet report...
I20260620 23:01:58.120292 29148 tablet_bootstrap.cc:492] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: No bootstrap required, opened a new log
I20260620 23:01:58.120363 29148 ts_tablet_manager.cc:1397] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 23:01:58.122227 29148 raft_consensus.cc:348] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.122376 29148 raft_consensus.cc:374] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:58.122398 29148 raft_consensus.cc:729] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce78aa9ed2f745968b8f29b347691907, State: Initialized, Role: FOLLOWER
I20260620 23:01:58.122512 29148 consensus_queue.cc:260] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.122797 29148 ts_tablet_manager.cc:1428] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:01:58.122849 28986 heartbeater.cc:503] Master 127.21.66.126:34795 was elected leader, sending a full tablet report...
W20260620 23:01:58.179879 28989 tablet.cc:2369] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:01:58.279845 28852 tablet.cc:2369] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:01:58.310096 29155 raft_consensus.cc:482] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:01:58.310369 29155 raft_consensus.cc:504] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.310909 29155 leader_election.cc:288] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565), ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:01:58.314147 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.311096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:38565 (local address 127.21.66.65:48815)
0620 23:01:58.311305 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.311319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.311391 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.311627 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.311628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.311630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:58.311689 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.311690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.312510 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.312511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.313290 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.313292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.313848 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.313849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.313923 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.313926 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.314002 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.314046 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260620 23:01:58.314146 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.311189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:48815 (local address 127.21.66.67:38565)
0620 23:01:58.311241 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.311243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.311382 (+ 139us) server_negotiation.cc:407] Connection header received
0620 23:01:58.311474 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.311476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.311485 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.311537 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:58.311775 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.312430 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.313400 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.313766 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.313849 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.313858 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.314010 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.314054 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:01:58.314266 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.311287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:35233 (local address 127.21.66.66:36603)
0620 23:01:58.311337 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.311340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.311366 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:01:58.311398 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.311399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.311438 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.311490 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:58.311805 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.312461 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.313448 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.313838 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.313940 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.313949 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.314141 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.314183 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:01:58.314275 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.311093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:36603 (local address 127.21.66.65:35233)
0620 23:01:58.311220 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:58.311232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:58.311317 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:58.311534 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:58.311535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:58.311537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:58.311692 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.311694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.312555 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.312556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.313346 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:58.313348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.313939 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.313940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:58.314018 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.314026 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:01:58.314110 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:01:58.314184 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260620 23:01:58.315004 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22beeac08d694eaa9d037e7a73963eb8" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" is_pre_election: true
I20260620 23:01:58.315089 28940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22beeac08d694eaa9d037e7a73963eb8" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce78aa9ed2f745968b8f29b347691907" is_pre_election: true
I20260620 23:01:58.315167 29076 raft_consensus.cc:2454] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 0.
I20260620 23:01:58.315284 28940 raft_consensus.cc:2454] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 0.
I20260620 23:01:58.315392 28739 leader_election.cc:302] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc8d4962f326499fb2dd201cc3c1ef88, d2bbd190c5d24aeea1c0eb118d633f6a; no voters:
I20260620 23:01:58.315575 29155 raft_consensus.cc:2781] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:01:58.315649 29155 raft_consensus.cc:482] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:01:58.315680 29155 raft_consensus.cc:3037] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:58.316403 29155 raft_consensus.cc:504] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.316565 29155 leader_election.cc:288] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 election: Requested vote from peers d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565), ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:01:58.316707 28940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22beeac08d694eaa9d037e7a73963eb8" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce78aa9ed2f745968b8f29b347691907"
I20260620 23:01:58.316805 28940 raft_consensus.cc:3037] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:58.316839 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22beeac08d694eaa9d037e7a73963eb8" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a"
I20260620 23:01:58.316905 29076 raft_consensus.cc:3037] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:01:58.317739 29076 raft_consensus.cc:2454] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 1.
I20260620 23:01:58.317739 28940 raft_consensus.cc:2454] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 1.
I20260620 23:01:58.317936 28737 leader_election.cc:302] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc8d4962f326499fb2dd201cc3c1ef88, ce78aa9ed2f745968b8f29b347691907; no voters:
I20260620 23:01:58.318051 29155 raft_consensus.cc:2781] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:01:58.318188 29155 raft_consensus.cc:686] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 LEADER]: Becoming Leader. State: Replica: bc8d4962f326499fb2dd201cc3c1ef88, State: Running, Role: LEADER
I20260620 23:01:58.318297 29155 consensus_queue.cc:237] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } }
I20260620 23:01:58.319160 27885 catalog_manager.cc:5495] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc8d4962f326499fb2dd201cc3c1ef88 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } health_report { overall_health: UNKNOWN } } }
W20260620 23:01:58.322271 29125 tablet.cc:2369] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:01:58.335822 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.332930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.21.66.65:41777)
0620 23:01:58.332993 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.332996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.333004 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:58.333073 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.333074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.333092 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.333155 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:01:58.333369 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.333900 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.334824 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.335060 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.335138 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.335332 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:01:58.335443 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:01:58.335601 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.335662 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.335686 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:01:58.338188 28940 raft_consensus.cc:1261] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Refusing update from remote peer bc8d4962f326499fb2dd201cc3c1ef88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:58.338188 29076 raft_consensus.cc:1261] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Refusing update from remote peer bc8d4962f326499fb2dd201cc3c1ef88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:01:58.338562 29159 consensus_queue.cc:1035] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:58.338725 29155 consensus_queue.cc:1035] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:01:58.343519 29164 mvcc.cc:205] Tried to move back new op lower bound from 7299057739109277696 to 7299057739032281088. Current Snapshot: MvccSnapshot[applied={T|T < 7299057739109277696 or (T in {7299057739109277696})}]
I20260620 23:01:58.343659 29162 mvcc.cc:205] Tried to move back new op lower bound from 7299057739109277696 to 7299057739032281088. Current Snapshot: MvccSnapshot[applied={T|T < 7299057739109277696}]
I20260620 23:01:58.346727 27884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:58.346836 27884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:01:58.350034 27884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05367f8fccc5489692b9e4debe1b467f: Sending DeleteTablet for 3 replicas of tablet 22beeac08d694eaa9d037e7a73963eb8
I20260620 23:01:58.350504 28920 tablet_service.cc:1514] Processing DeleteTablet for tablet 22beeac08d694eaa9d037e7a73963eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:58 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:44562
I20260620 23:01:58.350531 28783 tablet_service.cc:1514] Processing DeleteTablet for tablet 22beeac08d694eaa9d037e7a73963eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:58 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:46322
I20260620 23:01:58.350553 29056 tablet_service.cc:1514] Processing DeleteTablet for tablet 22beeac08d694eaa9d037e7a73963eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:01:58 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:45306
I20260620 23:01:58.351035 29174 tablet_replica.cc:330] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: stopping tablet replica
I20260620 23:01:58.351187 29174 raft_consensus.cc:2229] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:58.351308 29174 raft_consensus.cc:2258] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:58.351876 29174 ts_tablet_manager.cc:1905] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:58.352861 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 27853
I20260620 23:01:58.352977 29172 tablet_replica.cc:330] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: stopping tablet replica
I20260620 23:01:58.353092 29172 raft_consensus.cc:2229] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:01:58.353230 29172 raft_consensus.cc:2258] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:58.354738 29174 ts_tablet_manager.cc:1918] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:58.354830 29174 log.cc:1196] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/22beeac08d694eaa9d037e7a73963eb8
I20260620 23:01:58.355166 29174 ts_tablet_manager.cc:1939] T 22beeac08d694eaa9d037e7a73963eb8 P d2bbd190c5d24aeea1c0eb118d633f6a: Deleting consensus metadata
I20260620 23:01:58.355389 29173 tablet_replica.cc:330] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: stopping tablet replica
I20260620 23:01:58.355527 29173 raft_consensus.cc:2229] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:01:58.356096 29173 raft_consensus.cc:2258] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:01:58.356675 29173 ts_tablet_manager.cc:1905] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:58.358845 29173 ts_tablet_manager.cc:1918] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:58.358929 29173 log.cc:1196] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/22beeac08d694eaa9d037e7a73963eb8
I20260620 23:01:58.359205 29173 ts_tablet_manager.cc:1939] T 22beeac08d694eaa9d037e7a73963eb8 P ce78aa9ed2f745968b8f29b347691907: Deleting consensus metadata
W20260620 23:01:58.359915 28875 connection.cc:535] server connection from 127.0.0.1:44562 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 23:01:58.360108 29011 connection.cc:535] server connection from 127.0.0.1:45306 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:01:58.360932 29172 ts_tablet_manager.cc:1905] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:01:58.361928 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.351742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.21.66.124:46137)
0620 23:01:58.351782 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.351785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.351793 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:01:58.351817 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.351817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.351828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.351897 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.352770 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.353462 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.355854 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.356168 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.356214 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.358222 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.358224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.358224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.358233 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.360079 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.360925 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.360927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.360927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.360989 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.361280 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.361281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.361281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.361409 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.361479 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.361710 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.361809 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:01:58.362887 29172 ts_tablet_manager.cc:1918] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:01:58.362977 29172 log.cc:1196] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/22beeac08d694eaa9d037e7a73963eb8
I20260620 23:01:58.363324 29172 ts_tablet_manager.cc:1939] T 22beeac08d694eaa9d037e7a73963eb8 P bc8d4962f326499fb2dd201cc3c1ef88: Deleting consensus metadata
W20260620 23:01:58.363788 28739 connection.cc:441] server connection from 127.0.0.1:46322 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46322 (request call id 1) could send its response
I20260620 23:01:58.371452 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.362932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.21.66.125:44163)
0620 23:01:58.363006 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.363008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.363415 (+ 407us) server_negotiation.cc:407] Connection header received
0620 23:01:58.363543 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.363545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.363555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.363586 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.364347 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.365235 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.365983 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.366327 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.366394 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.368148 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.368150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.368150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.368159 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.369943 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.370504 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.370505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.370506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.370567 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.370771 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.370773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.370773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.370946 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.371054 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.371242 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.371336 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:01:58.371618 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:58.362493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.21.66.124:46137)
0620 23:01:58.362589 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:58.362590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:58.362860 (+ 270us) server_negotiation.cc:407] Connection header received
0620 23:01:58.362973 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:58.362974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:58.362985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:58.363019 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:01:58.364281 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.365119 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.366238 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:58.366544 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:58.366618 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:58.368353 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:01:58.368354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:01:58.368355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:01:58.368363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:01:58.370084 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.370642 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.370643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.370643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.370713 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:01:58.370958 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:01:58.370959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:01:58.370959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:01:58.371095 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:01:58.371166 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:58.371420 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 23:01:58.371503 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:01:59.344024 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.343697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.66:56061)
0620 23:01:59.343830 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.343965 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
W20260620 23:01:59.344229 28986 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:34795 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
I20260620 23:01:59.344540 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.344178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:47819)
0620 23:01:59.344438 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.344500 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":64,"threads_started":1}
I20260620 23:01:59.344544 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.344271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.67:42799)
0620 23:01:59.344426 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.344500 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":44,"threads_started":1}
W20260620 23:01:59.344763 28849 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:34795 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
W20260620 23:01:59.345175 29122 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:34795 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
I20260620 23:01:59.851416 29186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05367f8fccc5489692b9e4debe1b467f)
I20260620 23:01:59.851646 29187 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05367f8fccc5489692b9e4debe1b467f)
I20260620 23:01:59.851621 29186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } }
I20260620 23:01:59.851773 29187 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } }
I20260620 23:01:59.851980 29186 leader_election.cc:288] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05367f8fccc5489692b9e4debe1b467f (127.21.66.126:34795), cfd59e730385467685ff491c0ff20e3c (127.21.66.124:46137)
I20260620 23:01:59.852044 29187 leader_election.cc:288] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05367f8fccc5489692b9e4debe1b467f (127.21.66.126:34795), 7ca6226bb5854c73a510b88457ae8789 (127.21.66.125:44163)
I20260620 23:01:59.852365 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.852077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41754)
0620 23:01:59.852271 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.852323 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":54,"threads_started":1}
I20260620 23:01:59.852389 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.852182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41768)
0620 23:01:59.852316 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.852354 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":52,"threads_started":1}
W20260620 23:01:59.852483 28655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
W20260620 23:01:59.852535 28468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
I20260620 23:01:59.853050 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.852852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41772)
0620 23:01:59.852961 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.852996 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260620 23:01:59.853108 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.852990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41784)
0620 23:01:59.853048 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.853078 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 23:01:59.853138 28655 leader_election.cc:334] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05367f8fccc5489692b9e4debe1b467f (127.21.66.126:34795): Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
W20260620 23:01:59.853251 28468 leader_election.cc:334] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 05367f8fccc5489692b9e4debe1b467f (127.21.66.126:34795): Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
I20260620 23:01:59.855898 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.853041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.21.66.124:46137)
0620 23:01:59.853168 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:59.853170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:59.853179 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:59.853205 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:59.853205 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:59.853216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:59.853293 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:59.853496 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.854066 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.855155 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.855607 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.855658 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.855668 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:59.855821 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 23:01:59.855851 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 23:01:59.855942 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.852146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:46137 (local address 127.0.0.1:53064)
0620 23:01:59.852298 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.852322 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:59.852381 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:59.853299 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:59.853339 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:59.853342 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:59.853397 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.853404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.854170 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.854171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.855073 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.855075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.855678 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.855679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.855738 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.855742 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:59.855813 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:01:59.855853 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260620 23:01:59.856287 28680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ca6226bb5854c73a510b88457ae8789" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cfd59e730385467685ff491c0ff20e3c" is_pre_election: true
I20260620 23:01:59.856406 28680 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ca6226bb5854c73a510b88457ae8789 in term 1.
I20260620 23:01:59.856434 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.852398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.21.66.125:44163)
0620 23:01:59.852490 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:59.852493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:59.853072 (+ 579us) server_negotiation.cc:407] Connection header received
0620 23:01:59.853181 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:59.853182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:59.853193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:59.853235 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:59.853471 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.854232 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.855384 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.856048 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.856110 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.856122 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:59.856314 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:01:59.856359 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":68,"threads_started":1}
I20260620 23:01:59.856405 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.852773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44163 (local address 127.0.0.1:35096)
0620 23:01:59.852954 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.852964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:59.853102 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:59.853306 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:59.853307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:59.853339 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:59.853392 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.853394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.854312 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.854313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.855295 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.855297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.856162 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.856163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.856235 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.856238 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:01:59.856302 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:01:59.856336 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 23:01:59.856591 28467 leader_election.cc:302] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ca6226bb5854c73a510b88457ae8789, cfd59e730385467685ff491c0ff20e3c; no voters: 05367f8fccc5489692b9e4debe1b467f
I20260620 23:01:59.856689 29186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:01:59.856735 29186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 05367f8fccc5489692b9e4debe1b467f)
I20260620 23:01:59.856771 29186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:01:59.856873 28493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfd59e730385467685ff491c0ff20e3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7ca6226bb5854c73a510b88457ae8789" is_pre_election: true
I20260620 23:01:59.857556 29186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } }
I20260620 23:01:59.857734 28493 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cfd59e730385467685ff491c0ff20e3c in current term 2: Already voted for candidate 7ca6226bb5854c73a510b88457ae8789 in this term.
I20260620 23:01:59.857744 29186 leader_election.cc:288] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [CANDIDATE]: Term 2 election: Requested vote from peers 05367f8fccc5489692b9e4debe1b467f (127.21.66.126:34795), cfd59e730385467685ff491c0ff20e3c (127.21.66.124:46137)
I20260620 23:01:59.857913 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.857826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41790)
0620 23:01:59.857859 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.857892 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 23:01:59.857893 28680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ca6226bb5854c73a510b88457ae8789" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cfd59e730385467685ff491c0ff20e3c"
I20260620 23:01:59.857937 28657 leader_election.cc:302] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfd59e730385467685ff491c0ff20e3c; no voters: 05367f8fccc5489692b9e4debe1b467f, 7ca6226bb5854c73a510b88457ae8789
I20260620 23:01:59.857995 28680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:01:59.858426 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.858314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41802)
0620 23:01:59.858382 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.858408 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 23:01:59.858516 28468 leader_election.cc:334] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 05367f8fccc5489692b9e4debe1b467f (127.21.66.126:34795): Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
I20260620 23:01:59.858855 28680 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ca6226bb5854c73a510b88457ae8789 in term 2.
I20260620 23:01:59.858974 29187 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:01:59.859072 28467 leader_election.cc:302] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ca6226bb5854c73a510b88457ae8789, cfd59e730385467685ff491c0ff20e3c; no voters: 05367f8fccc5489692b9e4debe1b467f
I20260620 23:01:59.859177 29186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:01:59.859333 29186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [term 2 LEADER]: Becoming Leader. State: Replica: 7ca6226bb5854c73a510b88457ae8789, State: Running, Role: LEADER
I20260620 23:01:59.859431 29186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } }
I20260620 23:01:59.859828 29196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ca6226bb5854c73a510b88457ae8789. Latest consensus state: current_term: 2 leader_uuid: "7ca6226bb5854c73a510b88457ae8789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:59.859926 29196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:59.860109 29198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:01:59.860558 29198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:01:59.860623 29198 catalog_manager.cc:1259] Loaded cluster ID: daaf26df389f444f9e25258b158619b8
I20260620 23:01:59.860680 29198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:01:59.860882 29198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:01:59.860975 29198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Loaded TSK: 0
I20260620 23:01:59.861152 29198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:01:59.909562 28483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:01:59.911581 28483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:01:59.912871 28680 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [term 2 FOLLOWER]: Refusing update from remote peer 7ca6226bb5854c73a510b88457ae8789: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 23:01:59.912899 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.912722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41832)
0620 23:01:59.912813 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.912858 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 23:01:59.913218 29196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 23:01:59.913547 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.913369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.0.0.1:41840)
0620 23:01:59.913482 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.913514 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 23:01:59.913630 28468 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 -> Peer 05367f8fccc5489692b9e4debe1b467f (127.21.66.126:34795): Couldn't send request to peer 05367f8fccc5489692b9e4debe1b467f. Status: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:01:59.913918 29187 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ca6226bb5854c73a510b88457ae8789. Latest consensus state: current_term: 2 leader_uuid: "7ca6226bb5854c73a510b88457ae8789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:59.914021 29187 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:59.914490 29199 mvcc.cc:205] Tried to move back new op lower bound from 7299057745560563712 to 7299057745344524288. Current Snapshot: MvccSnapshot[applied={T|T < 7299057745560563712}]
I20260620 23:01:59.914598 29186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7ca6226bb5854c73a510b88457ae8789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:59.914693 29186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789 [sys.catalog]: This master's current role is: LEADER
I20260620 23:01:59.914817 29187 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7ca6226bb5854c73a510b88457ae8789" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "05367f8fccc5489692b9e4debe1b467f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34795 } } peers { permanent_uuid: "7ca6226bb5854c73a510b88457ae8789" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44163 } attrs { promote: false } } peers { permanent_uuid: "cfd59e730385467685ff491c0ff20e3c" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 46137 } attrs { promote: false } } }
I20260620 23:01:59.914942 29187 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd59e730385467685ff491c0ff20e3c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:01:59.920229 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.916655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.21.66.65:41777)
0620 23:01:59.916768 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:59.916770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:59.916779 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:01:59.916852 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:59.916854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:59.916865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:59.916938 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:59.917087 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.917823 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.919319 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.919868 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.919945 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.919967 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:59.920098 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 23:01:59.920148 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260620 23:01:59.920241 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.916586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:41777 (local address 127.0.0.1:46346)
0620 23:01:59.916703 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.916715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:59.916788 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:59.916942 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:59.916944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:59.916947 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:59.917008 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.917011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.917908 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.917909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.919231 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.919233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.919949 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.919950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.920021 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.920025 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:01:59.920084 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:01:59.920141 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 23:01:59.920264 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.916605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:38565 (local address 127.0.0.1:45310)
0620 23:01:59.917022 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.917032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:59.917084 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:59.917264 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:59.917265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:59.917266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:59.917482 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.917484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.918288 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.918289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.919066 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.919068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.919989 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.919990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.920078 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.920083 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:59.920133 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:01:59.920174 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":292,"thread_start_us":104,"threads_started":1}
I20260620 23:01:59.920483 28783 tablet_service.cc:1467] Processing CreateTablet for tablet 117ddfaf6b404ddb87e6cb7069e656b2 (DEFAULT_TABLE table=test-table [id=68f75040af85470d8b8d6d462fe71730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:59.920619 28783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117ddfaf6b404ddb87e6cb7069e656b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:59.920814 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.916697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.21.66.67:38565)
0620 23:01:59.916847 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:59.916850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:59.917134 (+ 284us) server_negotiation.cc:407] Connection header received
0620 23:01:59.917161 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:59.917162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:59.917172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:59.917225 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:59.917556 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.918212 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.919166 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.919882 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.919943 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.919957 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:59.920647 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 23:01:59.920731 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":46,"threads_started":1}
I20260620 23:01:59.921015 29056 tablet_service.cc:1467] Processing CreateTablet for tablet 117ddfaf6b404ddb87e6cb7069e656b2 (DEFAULT_TABLE table=test-table [id=68f75040af85470d8b8d6d462fe71730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:59.921118 29056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117ddfaf6b404ddb87e6cb7069e656b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:59.922493 29208 tablet_bootstrap.cc:492] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88: Bootstrap starting.
I20260620 23:01:59.922605 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.916603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:36603 (local address 127.0.0.1:44574)
0620 23:01:59.917000 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.917011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:01:59.917058 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:01:59.917337 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:01:59.917338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:01:59.917339 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:01:59.917377 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.917378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.918323 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.918324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.919482 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:01:59.919483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.922322 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.922323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:01:59.922403 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.922408 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:01:59.922468 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:01:59.922520 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260620 23:01:59.922628 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.916720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.21.66.66:36603)
0620 23:01:59.916835 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 23:01:59.916838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:01:59.917053 (+ 215us) server_negotiation.cc:407] Connection header received
0620 23:01:59.917219 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:01:59.917221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:01:59.917233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:01:59.917337 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:01:59.917479 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.918216 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.919608 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:01:59.920339 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:01:59.920415 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:01:59.920429 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 23:01:59.922494 (+ 2065us) server_negotiation.cc:299] Negotiation successful
0620 23:01:59.922549 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260620 23:01:59.922802 29209 tablet_bootstrap.cc:492] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: Bootstrap starting.
I20260620 23:01:59.922971 28920 tablet_service.cc:1467] Processing CreateTablet for tablet 117ddfaf6b404ddb87e6cb7069e656b2 (DEFAULT_TABLE table=test-table [id=68f75040af85470d8b8d6d462fe71730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:01:59.923108 28920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 117ddfaf6b404ddb87e6cb7069e656b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:01:59.923208 29209 tablet_bootstrap.cc:654] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:59.923756 29209 tablet_bootstrap.cc:492] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: No bootstrap required, opened a new log
I20260620 23:01:59.923800 29209 ts_tablet_manager.cc:1397] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:59.923938 29209 raft_consensus.cc:348] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:01:59.924017 29209 raft_consensus.cc:374] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:59.924031 29209 raft_consensus.cc:729] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bbd190c5d24aeea1c0eb118d633f6a, State: Initialized, Role: FOLLOWER
I20260620 23:01:59.924105 29209 consensus_queue.cc:260] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:01:59.924261 29209 ts_tablet_manager.cc:1428] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:01:59.924403 29212 tablet_bootstrap.cc:492] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: Bootstrap starting.
I20260620 23:01:59.924713 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.924490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.67:47227)
0620 23:01:59.924631 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.924669 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260620 23:01:59.924815 29212 tablet_bootstrap.cc:654] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:59.925089 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.924910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.67:52711)
0620 23:01:59.925038 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.925063 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260620 23:01:59.925351 29208 tablet_bootstrap.cc:654] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88: Neither blocks nor log segments found. Creating new log.
I20260620 23:01:59.925534 29212 tablet_bootstrap.cc:492] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: No bootstrap required, opened a new log
I20260620 23:01:59.925575 29212 ts_tablet_manager.cc:1397] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:59.925724 29212 raft_consensus.cc:348] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:01:59.925808 29212 raft_consensus.cc:374] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:59.925829 29212 raft_consensus.cc:729] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce78aa9ed2f745968b8f29b347691907, State: Initialized, Role: FOLLOWER
I20260620 23:01:59.925907 29212 consensus_queue.cc:260] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:01:59.925994 29208 tablet_bootstrap.cc:492] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88: No bootstrap required, opened a new log
I20260620 23:01:59.926047 29208 ts_tablet_manager.cc:1397] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 23:01:59.926085 29212 ts_tablet_manager.cc:1428] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:01:59.926431 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.926236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.66:54871)
0620 23:01:59.926345 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.926378 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":75,"threads_started":1}
I20260620 23:01:59.926729 29208 raft_consensus.cc:348] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:01:59.926820 29208 raft_consensus.cc:374] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:01:59.926846 29208 raft_consensus.cc:729] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8d4962f326499fb2dd201cc3c1ef88, State: Initialized, Role: FOLLOWER
I20260620 23:01:59.926903 29208 consensus_queue.cc:260] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:01:59.926941 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.926801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.66:43467)
0620 23:01:59.926869 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.926917 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 23:01:59.927054 29208 ts_tablet_manager.cc:1428] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:01:59.927413 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.927201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:38995)
0620 23:01:59.927347 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.927381 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":110,"threads_started":1}
I20260620 23:01:59.927765 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:01:59.927666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:46289)
0620 23:01:59.927702 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 23:01:59.927736 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 23:02:00.064426 29219 raft_consensus.cc:482] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:00.064543 29219 raft_consensus.cc:504] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:02:00.064770 29219 leader_election.cc:288] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565), ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:02:00.064967 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117ddfaf6b404ddb87e6cb7069e656b2" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" is_pre_election: true
I20260620 23:02:00.065078 29076 raft_consensus.cc:2454] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 0.
I20260620 23:02:00.065032 28940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117ddfaf6b404ddb87e6cb7069e656b2" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce78aa9ed2f745968b8f29b347691907" is_pre_election: true
I20260620 23:02:00.065181 28940 raft_consensus.cc:2454] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 0.
I20260620 23:02:00.065248 28739 leader_election.cc:302] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc8d4962f326499fb2dd201cc3c1ef88, d2bbd190c5d24aeea1c0eb118d633f6a; no voters:
I20260620 23:02:00.065388 29219 raft_consensus.cc:2781] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:00.065451 29219 raft_consensus.cc:482] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:00.065475 29219 raft_consensus.cc:3037] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:00.066341 29219 raft_consensus.cc:504] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:02:00.066501 29219 leader_election.cc:288] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 election: Requested vote from peers d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565), ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603)
I20260620 23:02:00.066718 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117ddfaf6b404ddb87e6cb7069e656b2" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a"
I20260620 23:02:00.066819 29076 raft_consensus.cc:3037] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:00.066821 28940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "117ddfaf6b404ddb87e6cb7069e656b2" candidate_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce78aa9ed2f745968b8f29b347691907"
I20260620 23:02:00.066934 28940 raft_consensus.cc:3037] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:00.067492 29076 raft_consensus.cc:2454] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 1.
I20260620 23:02:00.067626 28940 raft_consensus.cc:2454] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc8d4962f326499fb2dd201cc3c1ef88 in term 1.
I20260620 23:02:00.067741 28739 leader_election.cc:302] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc8d4962f326499fb2dd201cc3c1ef88, d2bbd190c5d24aeea1c0eb118d633f6a; no voters:
I20260620 23:02:00.067860 29219 raft_consensus.cc:2781] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:00.068030 29219 raft_consensus.cc:686] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [term 1 LEADER]: Becoming Leader. State: Replica: bc8d4962f326499fb2dd201cc3c1ef88, State: Running, Role: LEADER
I20260620 23:02:00.068104 29219 consensus_queue.cc:237] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } }
I20260620 23:02:00.068205 28851 heartbeater.cc:503] Master 127.21.66.125:44163 was elected leader, sending a full tablet report...
I20260620 23:02:00.068549 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.068315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:43287)
0620 23:02:00.068360 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.068401 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 23:02:00.068790 28483 catalog_manager.cc:5495] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc8d4962f326499fb2dd201cc3c1ef88 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc8d4962f326499fb2dd201cc3c1ef88" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 41777 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:00.104254 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.101744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.21.66.65:41777)
0620 23:02:00.101807 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:00.101809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:00.101821 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:02:00.101844 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:00.101845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:00.101853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:00.101907 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:00.102094 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:00.102602 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:00.103397 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:00.103605 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:00.103662 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:00.103880 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:00.103971 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:00.104022 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:00.104159 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 23:02:00.104183 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:02:00.105362 29076 raft_consensus.cc:1261] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Refusing update from remote peer bc8d4962f326499fb2dd201cc3c1ef88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:00.105362 28940 raft_consensus.cc:1261] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Refusing update from remote peer bc8d4962f326499fb2dd201cc3c1ef88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:00.105718 29221 consensus_queue.cc:1035] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bbd190c5d24aeea1c0eb118d633f6a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:00.105863 29219 consensus_queue.cc:1035] T 117ddfaf6b404ddb87e6cb7069e656b2 P bc8d4962f326499fb2dd201cc3c1ef88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce78aa9ed2f745968b8f29b347691907" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:00.106330 29124 heartbeater.cc:503] Master 127.21.66.125:44163 was elected leader, sending a full tablet report...
I20260620 23:02:00.106801 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.106599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.67:34855)
0620 23:02:00.106712 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.106757 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 23:02:00.106946 28988 heartbeater.cc:503] Master 127.21.66.125:44163 was elected leader, sending a full tablet report...
I20260620 23:02:00.107333 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.107211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.66:50567)
0620 23:02:00.107251 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.107297 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 23:02:00.108107 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.107970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.67:44227)
0620 23:02:00.108026 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.108068 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:02:00.108278 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.108111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:36907)
0620 23:02:00.108210 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.108243 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 23:02:00.108686 29220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.108548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.65:41121)
0620 23:02:00.108616 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.108657 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:02:00.108686 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.108504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34795 (local address 127.21.66.66:58273)
0620 23:02:00.108623 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.108652 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 23:02:00.110893 28483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:00.110978 28483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:00.112293 28483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ca6226bb5854c73a510b88457ae8789: Sending DeleteTablet for 3 replicas of tablet 117ddfaf6b404ddb87e6cb7069e656b2
I20260620 23:02:00.112655 29056 tablet_service.cc:1514] Processing DeleteTablet for tablet 117ddfaf6b404ddb87e6cb7069e656b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:00 UTC) from {username='oryx', principal='oryx/127.21.66.125@KRBTEST.COM'} at 127.0.0.1:45310
I20260620 23:02:00.112663 28920 tablet_service.cc:1514] Processing DeleteTablet for tablet 117ddfaf6b404ddb87e6cb7069e656b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:00 UTC) from {username='oryx', principal='oryx/127.21.66.125@KRBTEST.COM'} at 127.0.0.1:44574
I20260620 23:02:00.112692 28783 tablet_service.cc:1514] Processing DeleteTablet for tablet 117ddfaf6b404ddb87e6cb7069e656b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:00 UTC) from {username='oryx', principal='oryx/127.21.66.125@KRBTEST.COM'} at 127.0.0.1:46346
I20260620 23:02:00.112879 29235 tablet_replica.cc:330] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: stopping tablet replica
I20260620 23:02:00.112958 29235 raft_consensus.cc:2229] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:00.113013 29235 raft_consensus.cc:2258] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:00.113090 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 28630
I20260620 23:02:00.113363 29235 ts_tablet_manager.cc:1905] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:00.115033 29234 tablet_replica.cc:330] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: stopping tablet replica
I20260620 23:02:00.115105 29234 raft_consensus.cc:2229] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:00.115159 29234 raft_consensus.cc:2258] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:00.116928 29235 ts_tablet_manager.cc:1918] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:00.116981 29234 ts_tablet_manager.cc:1905] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:00.116997 29235 log.cc:1196] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/117ddfaf6b404ddb87e6cb7069e656b2
I20260620 23:02:00.117261 29235 ts_tablet_manager.cc:1939] T 117ddfaf6b404ddb87e6cb7069e656b2 P d2bbd190c5d24aeea1c0eb118d633f6a: Deleting consensus metadata
I20260620 23:02:00.117866 28470 catalog_manager.cc:4841] TS d2bbd190c5d24aeea1c0eb118d633f6a (127.21.66.67:38565): tablet 117ddfaf6b404ddb87e6cb7069e656b2 (table test-table [id=68f75040af85470d8b8d6d462fe71730]) successfully deleted
I20260620 23:02:00.118628 29234 ts_tablet_manager.cc:1918] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:00.118680 29234 log.cc:1196] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/117ddfaf6b404ddb87e6cb7069e656b2
I20260620 23:02:00.118960 29234 ts_tablet_manager.cc:1939] T 117ddfaf6b404ddb87e6cb7069e656b2 P ce78aa9ed2f745968b8f29b347691907: Deleting consensus metadata
I20260620 23:02:00.119318 28468 catalog_manager.cc:4841] TS ce78aa9ed2f745968b8f29b347691907 (127.21.66.66:36603): tablet 117ddfaf6b404ddb87e6cb7069e656b2 (table test-table [id=68f75040af85470d8b8d6d462fe71730]) successfully deleted
I20260620 23:02:00.121604 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:00.121478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:41777 (local address 127.0.0.1:46360)
0620 23:02:00.121526 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:00.121561 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:41777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 23:02:00.121778 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 28857
W20260620 23:02:00.124962 28469 catalog_manager.cc:4584] TS bc8d4962f326499fb2dd201cc3c1ef88 (127.21.66.65:41777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 117ddfaf6b404ddb87e6cb7069e656b2: Network error: Client connection negotiation failed: client connection to 127.21.66.65:41777: connect: Connection refused (error 111)
I20260620 23:02:00.128860 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 28993
I20260620 23:02:00.134070 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 28349
I20260620 23:02:00.139653 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 28537
2026-06-20T23:02:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4984 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:02:00 dist-test-slave-qgtk krb5kdc[29239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:02:00 dist-test-slave-qgtk krb5kdc[29239](info): set up 2 sockets
Jun 20 23:02:00 dist-test-slave-qgtk krb5kdc[29239](info): commencing operation
krb5kdc: starting...
W20260620 23:02:01.202935 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab.
Entry for principal oryx/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab.
I20260620 23:02:01.332610 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:42301
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab
--principal=oryx/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:42301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:01.415997 29252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:01.416280 29252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:01.416319 29252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:01.417583 29252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:01.417672 29252 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:02:01.417698 29252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:01.417728 29252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:01.417761 29252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:01.417784 29252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:01.419428 29252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:42301
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.126.keytab
--principal=oryx/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:42301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:01.419745 29252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:01.420001 29252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:01.425606 29258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.425597 29260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.425597 29257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:01.426040 29252 server_base.cc:1034] running on GCE node
I20260620 23:02:01.426329 29252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:01.426694 29252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:01.427901 29252 hybrid_clock.cc:648] HybridClock initialized: now 1781996521427876 us; error 44 us; skew 500 ppm
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:01.434677 29252 init.cc:377] Logged in from keytab as oryx/127.21.66.126@KRBTEST.COM (short username oryx)
I20260620 23:02:01.435287 29252 webserver.cc:460] Webserver started at http://127.21.66.126:43117/ using document root <none> and password file <none>
I20260620 23:02:01.435571 29252 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:01.435649 29252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:01.435804 29252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:01.436973 29252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "b6372c8dfabc4d40b19a4726e76d5b54"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "e0a4457ed3cb0a38ec7b7a79cdd74698"
server_key_iv: "9c0d6ff63751f41af588d9d36b9ecbce"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.437376 29252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "b6372c8dfabc4d40b19a4726e76d5b54"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "e0a4457ed3cb0a38ec7b7a79cdd74698"
server_key_iv: "9c0d6ff63751f41af588d9d36b9ecbce"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.438958 29252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:02:01.439867 29267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:01.440068 29252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:01.440186 29252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "b6372c8dfabc4d40b19a4726e76d5b54"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "e0a4457ed3cb0a38ec7b7a79cdd74698"
server_key_iv: "9c0d6ff63751f41af588d9d36b9ecbce"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.440267 29252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:01.458220 29252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:01.460681 29252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:01.460861 29252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:01.465848 29252 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:42301
I20260620 23:02:01.465898 29329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:42301 every 8 connection(s)
I20260620 23:02:01.466310 29252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:02:01.467001 29330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:01.468082 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 29252
I20260620 23:02:01.468250 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:02:01.468446 21769 external_mini_cluster.cc:1428] Setting key ca8e6f54f9e12012c6515053e7fd6cb2
I20260620 23:02:01.469944 29330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Bootstrap starting.
I20260620 23:02:01.470646 29330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:01.470957 29330 log.cc:824] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:01.471823 29330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: No bootstrap required, opened a new log
I20260620 23:02:01.474066 29330 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42301 } }
I20260620 23:02:01.474330 29330 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:01.474390 29330 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6372c8dfabc4d40b19a4726e76d5b54, State: Initialized, Role: FOLLOWER
I20260620 23:02:01.474568 29330 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42301 } }
I20260620 23:02:01.474666 29330 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:01.474718 29330 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:01.474781 29330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:02:01.475692 29330 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42301 } }
I20260620 23:02:01.475850 29330 leader_election.cc:302] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6372c8dfabc4d40b19a4726e76d5b54; no voters:
I20260620 23:02:01.476084 29330 leader_election.cc:288] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:01.476109 29335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:01.476260 29335 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [term 1 LEADER]: Becoming Leader. State: Replica: b6372c8dfabc4d40b19a4726e76d5b54, State: Running, Role: LEADER
I20260620 23:02:01.476372 29335 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42301 } }
I20260620 23:02:01.476419 29330 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:01.476984 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42301 } } }
I20260620 23:02:01.477065 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:01.477592 29337 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6372c8dfabc4d40b19a4726e76d5b54. Latest consensus state: current_term: 1 leader_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6372c8dfabc4d40b19a4726e76d5b54" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42301 } } }
I20260620 23:02:01.477710 29337 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:01.478772 29343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:01.479493 29343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:01.480827 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.469009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.21.66.126:42301)
0620 23:02:01.469197 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:01.469207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:01.469245 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:02:01.469796 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:01.469804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:01.469857 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:01.470017 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:01.470720 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.471890 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.472670 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.473081 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.473201 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.476155 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:01.476160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:01.476160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:01.476175 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:01.479262 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.479621 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.479624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.479625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.479724 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.479937 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.479938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.479938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.480184 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:01.480340 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:01.480440 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 23:02:01.480582 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260620 23:02:01.481164 29343 catalog_manager.cc:1347] Generated new cluster ID: f7345da315994a5789252f9c71b5f439
I20260620 23:02:01.481227 29343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:01.489455 29343 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:01.490083 29343 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:01.495373 29343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Generated new TSK 0
I20260620 23:02:01.495585 29343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab.
Entry for principal oryx/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab.
I20260620 23:02:01.531663 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42301
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab
--principal=oryx/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:01.646059 29356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:01.646406 29356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:01.646454 29356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:01.647933 29356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:01.648007 29356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:01.648047 29356 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:02:01.648097 29356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:02:01.649883 29356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab
--principal=oryx/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:01.650223 29356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:01.650486 29356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:01.656090 29362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.656090 29361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.656121 29364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:01.656507 29356 server_base.cc:1034] running on GCE node
I20260620 23:02:01.656774 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:01.657058 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:01.658200 29356 hybrid_clock.cc:648] HybridClock initialized: now 1781996521658157 us; error 72 us; skew 500 ppm
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:01.660871 29356 init.cc:377] Logged in from keytab as oryx/127.21.66.65@KRBTEST.COM (short username oryx)
I20260620 23:02:01.661409 29356 webserver.cc:460] Webserver started at http://127.21.66.65:45189/ using document root <none> and password file <none>
I20260620 23:02:01.661650 29356 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:01.661724 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:01.661890 29356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:01.663278 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "b98e52dc39580ccbc6801f5a33a0de24"
server_key_iv: "3af41c653357826afd4736fb2f17fd8d"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.663714 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "b98e52dc39580ccbc6801f5a33a0de24"
server_key_iv: "3af41c653357826afd4736fb2f17fd8d"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.665172 29356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:02:01.666046 29371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:01.666302 29356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:01.666447 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "b98e52dc39580ccbc6801f5a33a0de24"
server_key_iv: "3af41c653357826afd4736fb2f17fd8d"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.666534 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:01.676615 29356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:01.679473 29356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:01.679613 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:01.680190 29356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:01.680267 29356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:01.680310 29356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:01.680341 29356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:01.688254 29356 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:37291
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:02:01.688342 29486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:37291 every 8 connection(s)
I20260620 23:02:01.688727 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:02:01.692003 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.680635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:37277 (local address 127.21.66.126:42301)
0620 23:02:01.680694 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:01.680697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:01.680953 (+ 256us) server_negotiation.cc:407] Connection header received
0620 23:02:01.681461 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:01.681463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:01.681477 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:01.681574 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:01.682577 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.683545 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.685242 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.685656 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.685746 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.688865 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:01.688867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:01.688867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:01.688875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:01.690552 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.690998 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.690999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.691000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.691059 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.691332 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.691333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.691333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.691457 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:01.691533 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:01.691812 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 23:02:01.691903 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:02:01.692220 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.680631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42301 (local address 127.21.66.65:37277)
0620 23:02:01.680800 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:01.680850 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:01.681359 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:01.681625 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:01.681629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:01.682063 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:01.682453 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.682461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.683671 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.683672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.685129 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.685132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.685797 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.685798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.685952 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.686678 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:01.686691 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:01.688741 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:01.690660 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.690667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.690669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.690904 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.691131 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.691132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.691132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.691251 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.691523 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:01.691530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:01.691680 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 23:02:01.691792 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:02:01.691968 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260620 23:02:01.692776 29487 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42301
I20260620 23:02:01.692865 29487 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:01.693055 29487 heartbeater.cc:511] Master 127.21.66.126:42301 requested a full tablet report, sending...
I20260620 23:02:01.693608 29284 ts_manager.cc:194] Registered new tserver with Master: 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291)
I20260620 23:02:01.694447 29284 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:37277
I20260620 23:02:01.698184 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 29356
I20260620 23:02:01.698302 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:02:01.698536 21769 external_mini_cluster.cc:1428] Setting key 93a478f6137226e1ecaa3570198af40e
I20260620 23:02:01.700943 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.694081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42301 (local address 127.21.66.65:44253)
0620 23:02:01.694273 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:01.694285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:01.694344 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:01.694474 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:01.694474 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:01.694694 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:01.694758 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.694760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.695995 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.695996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.696587 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.696589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.696995 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.696996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.697056 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.697554 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:01.697558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:01.697896 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:01.699787 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.699789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.699789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.700039 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.700233 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.700234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.700234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.700304 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.700635 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:01.700636 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:01.700685 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:02:01.700763 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:02:01.700860 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":132,"threads_started":1}
I20260620 23:02:01.701020 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.694167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:44253 (local address 127.21.66.126:42301)
0620 23:02:01.694244 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:01.694246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:01.694345 (+ 99us) server_negotiation.cc:407] Connection header received
0620 23:02:01.694401 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:01.694402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:01.694410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:01.694456 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:01.695348 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.695884 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.696678 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.696903 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.696967 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.698036 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:01.698037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:01.698037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:01.698044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:01.699692 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.700114 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.700115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.700116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.700168 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.700393 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.700394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.700395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.700556 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:01.700635 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:01.700798 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:02:01.700886 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab.
Entry for principal oryx/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab.
I20260620 23:02:01.727553 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42301
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab
--principal=oryx/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:01.812703 29496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:01.812997 29496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:01.813037 29496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:01.814478 29496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:01.814525 29496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:01.814546 29496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:02:01.814564 29496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:02:01.816095 29496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab
--principal=oryx/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:01.816303 29496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:01.816500 29496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:01.821663 29501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.821635 29504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.821646 29502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:01.821899 29496 server_base.cc:1034] running on GCE node
I20260620 23:02:01.822261 29496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:01.822537 29496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:01.823745 29496 hybrid_clock.cc:648] HybridClock initialized: now 1781996521823718 us; error 35 us; skew 500 ppm
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:01.826476 29496 init.cc:377] Logged in from keytab as oryx/127.21.66.66@KRBTEST.COM (short username oryx)
I20260620 23:02:01.826969 29496 webserver.cc:460] Webserver started at http://127.21.66.66:40103/ using document root <none> and password file <none>
I20260620 23:02:01.827219 29496 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:01.827265 29496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:01.827451 29496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:01.828543 29496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "525252f8fbd94bf2960d2bd576251519"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "cc8d4415b69a0097b3f73d3cfd41cba5"
server_key_iv: "edc05ba6b7533acf36f4bc0d2af52d17"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.828899 29496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "525252f8fbd94bf2960d2bd576251519"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "cc8d4415b69a0097b3f73d3cfd41cba5"
server_key_iv: "edc05ba6b7533acf36f4bc0d2af52d17"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.830425 29496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 23:02:01.831296 29511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:01.831525 29496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:01.831600 29496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "525252f8fbd94bf2960d2bd576251519"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "cc8d4415b69a0097b3f73d3cfd41cba5"
server_key_iv: "edc05ba6b7533acf36f4bc0d2af52d17"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.831697 29496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:01.847935 29496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:01.850488 29496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:01.850644 29496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:01.851291 29496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:01.851344 29496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:01.851388 29496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:01.851408 29496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:02:01.858932 29496 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:36953
I20260620 23:02:01.859010 29626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:36953 every 8 connection(s)
I20260620 23:02:01.859304 29496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:02:01.862187 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.851806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:48249 (local address 127.21.66.126:42301)
0620 23:02:01.851894 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:01.851897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:01.852274 (+ 377us) server_negotiation.cc:407] Connection header received
0620 23:02:01.853039 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:01.853041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:01.853059 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:01.853161 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:01.854150 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.854682 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.856521 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.856747 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.856833 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.859386 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:01.859388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:01.859388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:01.859396 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:01.860714 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.861167 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.861168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.861169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.861240 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.861506 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.861507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.861507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.861630 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:01.861706 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:01.861988 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:02:01.862073 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:02:01.862385 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.851852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42301 (local address 127.21.66.66:48249)
0620 23:02:01.852066 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:01.852118 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:01.852883 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:01.853249 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:01.853255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:01.853677 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:01.854017 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.854024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.854822 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.854823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.856401 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.856405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.856842 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.856844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.857006 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.857598 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:01.857607 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:01.859196 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:01.860851 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.860858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.860861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.861087 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.861316 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.861317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.861318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.861432 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.861698 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:01.861705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:01.861850 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 23:02:01.861955 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 23:02:01.862115 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260620 23:02:01.862799 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 29496
I20260620 23:02:01.862875 29627 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42301
I20260620 23:02:01.862934 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:02:01.862970 29627 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:01.863168 21769 external_mini_cluster.cc:1428] Setting key e6a76e3f9cb02abd99dd1716d76be18f
I20260620 23:02:01.863168 29627 heartbeater.cc:511] Master 127.21.66.126:42301 requested a full tablet report, sending...
I20260620 23:02:01.863790 29284 ts_manager.cc:194] Registered new tserver with Master: 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953)
I20260620 23:02:01.864285 29284 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:48249
WARNING: no policy specified for oryx/127.21.66.67@KRBTEST.COM; defaulting to no policy
I20260620 23:02:01.872184 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.865269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42301 (local address 127.21.66.66:34375)
0620 23:02:01.865493 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:01.865507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:01.865582 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:01.865742 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:01.865743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:01.865970 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:01.866044 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.866046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.866772 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.866773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.867679 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:01.867682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.868141 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.868143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:01.868217 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.868685 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:01.868690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:01.869144 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:01.870976 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.870977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.870978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.871206 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.871406 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:01.871407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:01.871407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:01.871480 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:01.871824 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:01.871825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:01.871878 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:02:01.871961 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:02:01.872079 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260620 23:02:01.872181 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:01.865214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:34375 (local address 127.21.66.126:42301)
0620 23:02:01.865275 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:01.865293 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:01.865590 (+ 297us) server_negotiation.cc:407] Connection header received
0620 23:02:01.865645 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:01.865645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:01.865654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:01.865735 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:01.866165 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.866671 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.867837 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:01.868056 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:01.868113 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:01.869251 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:01.869253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:01.869253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:01.869262 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:01.870889 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.871283 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.871284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.871285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.871337 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:01.871549 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:01.871549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:01.871550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:01.871688 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:01.871827 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:01.872004 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 23:02:01.872079 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab.
Entry for principal oryx/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab.
I20260620 23:02:01.893656 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42301
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab
--principal=oryx/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:01.976893 29636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:01.977147 29636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:01.977190 29636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:01.978601 29636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:01.978668 29636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:01.978703 29636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:02:01.978742 29636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:02:01.980424 29636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab
--principal=oryx/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.66.126:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:01.980716 29636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:01.980962 29636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:01.986212 29644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.986202 29641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:01.986207 29642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:01.986444 29636 server_base.cc:1034] running on GCE node
I20260620 23:02:01.986673 29636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:01.986935 29636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:01.988111 29636 hybrid_clock.cc:648] HybridClock initialized: now 1781996521988094 us; error 32 us; skew 500 ppm
Jun 20 23:02:01 dist-test-slave-qgtk krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:01.991057 29636 init.cc:377] Logged in from keytab as oryx/127.21.66.67@KRBTEST.COM (short username oryx)
I20260620 23:02:01.991482 29636 webserver.cc:460] Webserver started at http://127.21.66.67:34483/ using document root <none> and password file <none>
I20260620 23:02:01.991694 29636 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:01.991739 29636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:01.991837 29636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:01.993098 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "40b79e6fd8d547d49e71f768c7ade66a"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "36884bd9badfaca59e54f1178c151147"
server_key_iv: "94ea49d84554b2fa80cd912456ec95eb"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.993525 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "40b79e6fd8d547d49e71f768c7ade66a"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "36884bd9badfaca59e54f1178c151147"
server_key_iv: "94ea49d84554b2fa80cd912456ec95eb"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.995098 29636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:02:01.996204 29651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:01.996480 29636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:01.996585 29636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "40b79e6fd8d547d49e71f768c7ade66a"
format_stamp: "Formatted at 2026-06-20 23:02:01 on dist-test-slave-qgtk"
server_key: "36884bd9badfaca59e54f1178c151147"
server_key_iv: "94ea49d84554b2fa80cd912456ec95eb"
server_key_version: "encryptionkey@0"
I20260620 23:02:01.996649 29636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:02.019333 29636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:02.023119 29636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:02.023301 29636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:02.024222 29636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:02.024283 29636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:02.024339 29636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:02.024358 29636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:02.032145 29636 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:33011
I20260620 23:02:02.032306 29766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:33011 every 8 connection(s)
I20260620 23:02:02.032496 29636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:02:02.036798 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.024739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:39141 (local address 127.21.66.126:42301)
0620 23:02:02.024788 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.024791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.025034 (+ 243us) server_negotiation.cc:407] Connection header received
0620 23:02:02.025882 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.025884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.025911 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.025965 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:02.026910 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.027806 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.029634 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.029976 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.030044 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.033191 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:02.033193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:02.033193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:02.033202 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:02.035108 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.035654 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.035655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.035656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.035724 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.035998 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.035999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.035999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.036134 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:02.036213 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.036591 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.036692 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:02:02.037024 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.024756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42301 (local address 127.21.66.67:39141)
0620 23:02:02.024912 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.024956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.025760 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.026048 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.026053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.026499 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:02.026804 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.026811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.027945 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.027946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.029511 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.029514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.030141 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.030142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.030312 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.031014 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:02.031028 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:02.033069 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:02.035202 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:02.035213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:02.035216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:02.035571 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:02.035798 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:02.035799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:02.035800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:02.035921 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:02.036203 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:02.036210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:02.036375 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.036568 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.036772 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260620 23:02:02.037554 29767 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42301
I20260620 23:02:02.037642 29767 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:02.037827 29767 heartbeater.cc:511] Master 127.21.66.126:42301 requested a full tablet report, sending...
I20260620 23:02:02.038211 29284 ts_manager.cc:194] Registered new tserver with Master: 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.038580 29284 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:39141
I20260620 23:02:02.039469 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 29636
I20260620 23:02:02.039585 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:02:02.039872 21769 external_mini_cluster.cc:1428] Setting key 1ca261f390f5868fb47edb3da63f3b6d
I20260620 23:02:02.040817 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:02.049088 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.038943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42301 (local address 127.21.66.67:37935)
0620 23:02:02.039142 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.039153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.039216 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.039415 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.039416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.039652 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:02.039719 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.039721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.040814 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.040815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.041640 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.041643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.042214 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.042215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.042294 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.042851 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:02.042857 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:02.043300 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:02.046010 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:02.046012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:02.046012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:02.046333 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:02.046579 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:02.046579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:02.046580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:02.046660 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:02.048595 (+ 1935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:02.048596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:02.048672 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.048800 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.048949 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260620 23:02:02.049152 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.039018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:37935 (local address 127.21.66.126:42301)
0620 23:02:02.039079 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.039082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.039207 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:02:02.039305 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.039306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.039322 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.039411 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:02.039808 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.040701 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.041741 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.042068 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.042109 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.043443 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:02.043444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:02.043444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:02.043453 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:02.045905 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.046428 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.046429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.046429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.046509 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.046742 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.046743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.046743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.046895 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:02.046965 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.048895 (+ 1930us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.049011 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:02:02.071015 29333 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:02.064246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.21.66.126:42301)
0620 23:02:02.064305 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.064307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.064322 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:02:02.064412 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.064413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.064424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.064510 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:02.065150 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.065675 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.066601 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.066814 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.066898 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.070897 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33710: BlockingRecv error: recv got EOF from 127.0.0.1:33710 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996522, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:02:02.080396 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.071828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.21.66.126:42301)
0620 23:02:02.071873 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.071875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.072027 (+ 152us) server_negotiation.cc:407] Connection header received
0620 23:02:02.072067 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.072068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.072083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.072162 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:02.072655 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.073522 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.074375 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.074754 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.074840 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.077554 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:02.077556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:02.077556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:02.077565 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:02.079227 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.079570 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.079572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.079572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.079634 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.079852 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.079853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.079853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.079978 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:02.080090 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.080194 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.080279 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:02:02.082590 29284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:02:02.084386 29284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:02.092442 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.087471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.21.66.65:37291)
0620 23:02:02.087717 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.087723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.087736 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:02:02.087780 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.087782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.087800 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.088044 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.088344 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.089475 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.091138 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.091822 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.091899 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.091980 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.092225 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.092271 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":155,"threads_started":1}
I20260620 23:02:02.092445 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.087405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33011 (local address 127.0.0.1:52348)
0620 23:02:02.088236 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.088246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.088302 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.088635 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.088637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.088639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.088675 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.088676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.089842 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.089843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.090905 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.090907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.091935 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.091936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.092082 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.092102 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.092199 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.092282 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":74,"thread_start_us":86,"threads_started":1}
I20260620 23:02:02.092456 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.087184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37291 (local address 127.0.0.1:43178)
0620 23:02:02.087549 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.087559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.087600 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.088148 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.088153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.088155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.088255 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.088258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.089567 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.089567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.091039 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.091042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.091936 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.091937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.092086 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.092102 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.092199 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.092371 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":149,"thread_start_us":90,"threads_started":1}
I20260620 23:02:02.092875 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.087972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52348 (local address 127.21.66.67:33011)
0620 23:02:02.088185 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.088189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.088360 (+ 171us) server_negotiation.cc:407] Connection header received
0620 23:02:02.088483 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.088486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.088500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.088571 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.088743 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.089751 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.090999 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.091801 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.091892 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.091984 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.092715 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.092760 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260620 23:02:02.093226 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.087184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:36953 (local address 127.0.0.1:60998)
0620 23:02:02.087369 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.087470 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.087554 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.088306 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.088307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.088309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.088363 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.088365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.089354 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.089356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.091040 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.091042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.092991 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.092992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.093057 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.093061 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.093111 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.093156 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":71,"thread_start_us":66,"threads_started":1}
I20260620 23:02:02.093495 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.087770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.21.66.66:36953)
0620 23:02:02.088003 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.088009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.088021 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:02:02.088063 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.088066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.088084 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.088175 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.088460 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.089243 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.092027 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.092789 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.092859 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.092954 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.093351 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.093396 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
I20260620 23:02:02.093631 29699 tablet_service.cc:1467] Processing CreateTablet for tablet a530ae5f059b46a5849f4f2bf72fb23a (DEFAULT_TABLE table=test-table [id=be069a2cf5db4d2ba9b7b01d406d055f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.094038 29699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a530ae5f059b46a5849f4f2bf72fb23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.094290 29558 tablet_service.cc:1467] Processing CreateTablet for tablet a530ae5f059b46a5849f4f2bf72fb23a (DEFAULT_TABLE table=test-table [id=be069a2cf5db4d2ba9b7b01d406d055f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.094796 29558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a530ae5f059b46a5849f4f2bf72fb23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.094889 29421 tablet_service.cc:1467] Processing CreateTablet for tablet a530ae5f059b46a5849f4f2bf72fb23a (DEFAULT_TABLE table=test-table [id=be069a2cf5db4d2ba9b7b01d406d055f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.095242 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a530ae5f059b46a5849f4f2bf72fb23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.097250 29795 tablet_bootstrap.cc:492] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Bootstrap starting.
I20260620 23:02:02.098039 29795 tablet_bootstrap.cc:654] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.098399 29795 log.cc:824] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:02.099242 29795 tablet_bootstrap.cc:492] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: No bootstrap required, opened a new log
I20260620 23:02:02.099319 29795 ts_tablet_manager.cc:1397] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 23:02:02.099596 29797 tablet_bootstrap.cc:492] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Bootstrap starting.
I20260620 23:02:02.099597 29798 tablet_bootstrap.cc:492] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Bootstrap starting.
I20260620 23:02:02.100277 29797 tablet_bootstrap.cc:654] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.100330 29798 tablet_bootstrap.cc:654] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.100651 29797 log.cc:824] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:02.100697 29798 log.cc:824] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:02.101570 29797 tablet_bootstrap.cc:492] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: No bootstrap required, opened a new log
I20260620 23:02:02.101643 29797 ts_tablet_manager.cc:1397] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:02:02.101737 29795 raft_consensus.cc:348] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.101886 29795 raft_consensus.cc:374] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.101915 29795 raft_consensus.cc:729] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b79e6fd8d547d49e71f768c7ade66a, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.102051 29795 consensus_queue.cc:260] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.102392 29795 ts_tablet_manager.cc:1428] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 23:02:02.102413 29767 heartbeater.cc:503] Master 127.21.66.126:42301 was elected leader, sending a full tablet report...
I20260620 23:02:02.102742 29798 tablet_bootstrap.cc:492] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: No bootstrap required, opened a new log
I20260620 23:02:02.102823 29798 ts_tablet_manager.cc:1397] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 23:02:02.103816 29797 raft_consensus.cc:348] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.103987 29797 raft_consensus.cc:374] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.104032 29797 raft_consensus.cc:729] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.104152 29797 consensus_queue.cc:260] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.104453 29797 ts_tablet_manager.cc:1428] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:02.104481 29487 heartbeater.cc:503] Master 127.21.66.126:42301 was elected leader, sending a full tablet report...
I20260620 23:02:02.104830 29798 raft_consensus.cc:348] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.105010 29798 raft_consensus.cc:374] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.105057 29798 raft_consensus.cc:729] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525252f8fbd94bf2960d2bd576251519, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.105178 29798 consensus_queue.cc:260] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.105492 29798 ts_tablet_manager.cc:1428] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:02.105571 29627 heartbeater.cc:503] Master 127.21.66.126:42301 was elected leader, sending a full tablet report...
W20260620 23:02:02.109938 29628 tablet.cc:2369] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:02:02.190028 29488 tablet.cc:2369] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:02:02.191908 29803 raft_consensus.cc:482] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:02.192051 29803 raft_consensus.cc:504] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.192544 29803 leader_election.cc:288] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.196981 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.192885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:35435 (local address 127.21.66.65:37291)
0620 23:02:02.192918 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.192919 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.192928 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:02.192969 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.192970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.192983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.193089 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.193461 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.194300 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.196006 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.196564 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.196645 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.196654 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.196865 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.196908 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 23:02:02.196981 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.192822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:49591 (local address 127.21.66.67:33011)
0620 23:02:02.192877 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.192879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.193054 (+ 175us) server_negotiation.cc:407] Connection header received
0620 23:02:02.193096 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.193098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.193108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.193205 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.193526 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.194300 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.196009 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.196564 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.196645 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.196654 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.196870 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.196908 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:02:02.197007 29520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.192760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37291 (local address 127.21.66.66:35435)
0620 23:02:02.192822 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.192847 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.192900 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.193251 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.193252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.193254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.193296 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.193298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.194564 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.194565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.195895 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.195898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.196660 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.196661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.196753 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.196758 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.196821 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.196893 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":49}
I20260620 23:02:02.197009 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.192760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33011 (local address 127.21.66.66:49591)
0620 23:02:02.192985 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.192994 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.193049 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.193212 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.193213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.193215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.193294 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.193296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.194564 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.194565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.195895 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.195898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.196660 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.196661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.196753 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.196758 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.196856 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.196893 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":92,"thread_start_us":143,"threads_started":1}
I20260620 23:02:02.197502 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530ae5f059b46a5849f4f2bf72fb23a" candidate_uuid: "525252f8fbd94bf2960d2bd576251519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" is_pre_election: true
I20260620 23:02:02.197528 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530ae5f059b46a5849f4f2bf72fb23a" candidate_uuid: "525252f8fbd94bf2960d2bd576251519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a" is_pre_election: true
I20260620 23:02:02.197683 29441 raft_consensus.cc:2454] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525252f8fbd94bf2960d2bd576251519 in term 0.
I20260620 23:02:02.197682 29721 raft_consensus.cc:2454] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525252f8fbd94bf2960d2bd576251519 in term 0.
I20260620 23:02:02.197927 29514 leader_election.cc:302] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b79e6fd8d547d49e71f768c7ade66a, 525252f8fbd94bf2960d2bd576251519; no voters:
I20260620 23:02:02.198061 29803 raft_consensus.cc:2781] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:02.198238 29803 raft_consensus.cc:482] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:02.198276 29803 raft_consensus.cc:3037] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.199527 29803 raft_consensus.cc:504] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.199738 29803 leader_election.cc:288] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [CANDIDATE]: Term 1 election: Requested vote from peers 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.199939 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530ae5f059b46a5849f4f2bf72fb23a" candidate_uuid: "525252f8fbd94bf2960d2bd576251519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2"
I20260620 23:02:02.200054 29441 raft_consensus.cc:3037] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.200039 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a530ae5f059b46a5849f4f2bf72fb23a" candidate_uuid: "525252f8fbd94bf2960d2bd576251519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a"
I20260620 23:02:02.200132 29721 raft_consensus.cc:3037] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.200981 29441 raft_consensus.cc:2454] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525252f8fbd94bf2960d2bd576251519 in term 1.
I20260620 23:02:02.200981 29721 raft_consensus.cc:2454] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525252f8fbd94bf2960d2bd576251519 in term 1.
I20260620 23:02:02.201196 29514 leader_election.cc:302] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b79e6fd8d547d49e71f768c7ade66a, 525252f8fbd94bf2960d2bd576251519; no voters:
I20260620 23:02:02.201303 29803 raft_consensus.cc:2781] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:02.201452 29803 raft_consensus.cc:686] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 1 LEADER]: Becoming Leader. State: Replica: 525252f8fbd94bf2960d2bd576251519, State: Running, Role: LEADER
I20260620 23:02:02.201568 29803 consensus_queue.cc:237] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.202534 29284 catalog_manager.cc:5495] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525252f8fbd94bf2960d2bd576251519 (127.21.66.66). New cstate: current_term: 1 leader_uuid: "525252f8fbd94bf2960d2bd576251519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:02.212467 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.209699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.21.66.66:36953)
0620 23:02:02.209771 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.209778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.209786 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:02.209820 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.209821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.209839 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.209904 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:02.210072 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.210628 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.211493 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.211695 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.211760 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.211974 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.212083 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.212155 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.212234 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.212254 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:02:02.214520 29721 raft_consensus.cc:1261] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Refusing update from remote peer 525252f8fbd94bf2960d2bd576251519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.214584 29441 raft_consensus.cc:1261] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Refusing update from remote peer 525252f8fbd94bf2960d2bd576251519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.214944 29805 consensus_queue.cc:1035] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.215126 29803 consensus_queue.cc:1035] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.220247 29284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:02.220374 29284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:02.221316 29284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Sending DeleteTablet for 3 replicas of tablet a530ae5f059b46a5849f4f2bf72fb23a
I20260620 23:02:02.221740 29699 tablet_service.cc:1514] Processing DeleteTablet for tablet a530ae5f059b46a5849f4f2bf72fb23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52348
I20260620 23:02:02.221709 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet a530ae5f059b46a5849f4f2bf72fb23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:43178
I20260620 23:02:02.221984 29818 tablet_replica.cc:330] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: stopping tablet replica
I20260620 23:02:02.222009 29558 tablet_service.cc:1514] Processing DeleteTablet for tablet a530ae5f059b46a5849f4f2bf72fb23a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:60998
I20260620 23:02:02.222100 29818 raft_consensus.cc:2229] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:02.222267 29818 raft_consensus.cc:2258] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.222512 29819 tablet_replica.cc:330] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: stopping tablet replica
I20260620 23:02:02.222636 29819 raft_consensus.cc:2229] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:02.222595 29284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:02:02.222765 29284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:02.222776 29819 raft_consensus.cc:2258] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.223276 29820 tablet_replica.cc:330] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: stopping tablet replica
I20260620 23:02:02.223394 29820 raft_consensus.cc:2229] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:02.223538 29820 raft_consensus.cc:2258] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.223527 29818 ts_tablet_manager.cc:1905] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.224174 29820 ts_tablet_manager.cc:1905] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.224630 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 20f449f188e54b1c932502499311e5bc (DEFAULT_TABLE table=test-table [id=34c706d9ce204da1b2353dc169c190d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.224994 29699 tablet_service.cc:1467] Processing CreateTablet for tablet 20f449f188e54b1c932502499311e5bc (DEFAULT_TABLE table=test-table [id=34c706d9ce204da1b2353dc169c190d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.225162 29699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f449f188e54b1c932502499311e5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.226325 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f449f188e54b1c932502499311e5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.226336 29558 tablet_service.cc:1467] Processing CreateTablet for tablet 20f449f188e54b1c932502499311e5bc (DEFAULT_TABLE table=test-table [id=34c706d9ce204da1b2353dc169c190d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.226451 29558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f449f188e54b1c932502499311e5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.227192 29795 tablet_bootstrap.cc:492] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Bootstrap starting.
I20260620 23:02:02.227702 29795 tablet_bootstrap.cc:654] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.227859 29798 tablet_bootstrap.cc:492] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: Bootstrap starting.
I20260620 23:02:02.227909 29797 tablet_bootstrap.cc:492] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: Bootstrap starting.
I20260620 23:02:02.228322 29798 tablet_bootstrap.cc:654] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.228370 29797 tablet_bootstrap.cc:654] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.228856 29820 ts_tablet_manager.cc:1918] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:02.228967 29820 log.cc:1196] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/a530ae5f059b46a5849f4f2bf72fb23a
I20260620 23:02:02.229033 29795 tablet_bootstrap.cc:492] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: No bootstrap required, opened a new log
I20260620 23:02:02.229099 29795 ts_tablet_manager.cc:1397] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:02:02.229332 29820 ts_tablet_manager.cc:1939] T a530ae5f059b46a5849f4f2bf72fb23a P 525252f8fbd94bf2960d2bd576251519: Deleting consensus metadata
I20260620 23:02:02.229300 29795 raft_consensus.cc:348] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.229420 29795 raft_consensus.cc:374] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.229456 29795 raft_consensus.cc:729] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b79e6fd8d547d49e71f768c7ade66a, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.229529 29795 consensus_queue.cc:260] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.229641 29795 ts_tablet_manager.cc:1428] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:02.229655 29818 ts_tablet_manager.cc:1918] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:02.229825 29818 log.cc:1196] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/a530ae5f059b46a5849f4f2bf72fb23a
I20260620 23:02:02.229818 29270 catalog_manager.cc:4841] TS 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953): tablet a530ae5f059b46a5849f4f2bf72fb23a (table test-table [id=be069a2cf5db4d2ba9b7b01d406d055f]) successfully deleted
I20260620 23:02:02.230269 29818 ts_tablet_manager.cc:1939] T a530ae5f059b46a5849f4f2bf72fb23a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting consensus metadata
I20260620 23:02:02.230319 29797 tablet_bootstrap.cc:492] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: No bootstrap required, opened a new log
I20260620 23:02:02.230433 29797 ts_tablet_manager.cc:1397] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 23:02:02.230634 29797 raft_consensus.cc:348] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.230741 29797 raft_consensus.cc:374] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.230775 29797 raft_consensus.cc:729] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.230839 29797 consensus_queue.cc:260] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.230993 29797 ts_tablet_manager.cc:1428] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:02:02.231041 29268 catalog_manager.cc:4841] TS 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291): tablet a530ae5f059b46a5849f4f2bf72fb23a (table test-table [id=be069a2cf5db4d2ba9b7b01d406d055f]) successfully deleted
I20260620 23:02:02.231336 29798 tablet_bootstrap.cc:492] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: No bootstrap required, opened a new log
I20260620 23:02:02.231400 29798 ts_tablet_manager.cc:1397] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 23:02:02.231595 29798 raft_consensus.cc:348] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.231650 29819 ts_tablet_manager.cc:1905] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.231724 29798 raft_consensus.cc:374] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.231770 29798 raft_consensus.cc:729] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525252f8fbd94bf2960d2bd576251519, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.231824 29798 consensus_queue.cc:260] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.231950 29798 ts_tablet_manager.cc:1428] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:02:02.234257 29819 ts_tablet_manager.cc:1918] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:02.234366 29819 log.cc:1196] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/a530ae5f059b46a5849f4f2bf72fb23a
I20260620 23:02:02.234678 29819 ts_tablet_manager.cc:1939] T a530ae5f059b46a5849f4f2bf72fb23a P 40b79e6fd8d547d49e71f768c7ade66a: Deleting consensus metadata
I20260620 23:02:02.236109 29270 catalog_manager.cc:4841] TS 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011): tablet a530ae5f059b46a5849f4f2bf72fb23a (table test-table [id=be069a2cf5db4d2ba9b7b01d406d055f]) successfully deleted
I20260620 23:02:02.266201 29802 raft_consensus.cc:482] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:02.266315 29802 raft_consensus.cc:504] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.266700 29802 leader_election.cc:288] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.270905 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.266985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:58405 (local address 127.21.66.66:36953)
0620 23:02:02.267058 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.267060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.267177 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:02:02.267277 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.267278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.267289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.267367 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.267585 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.268380 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.269940 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.270456 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.270538 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.270549 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.270764 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.270811 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:02:02.270968 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.267310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:53679 (local address 127.21.66.67:33011)
0620 23:02:02.267356 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.267357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.267472 (+ 115us) server_negotiation.cc:407] Connection header received
0620 23:02:02.267538 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.267539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.267549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.267604 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.267773 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.268273 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.269916 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.270622 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.270700 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.270709 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.270853 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.270895 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:02:02.271171 29824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.266924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:36953 (local address 127.21.66.65:58405)
0620 23:02:02.267111 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.267123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.267207 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.267425 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.267426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.267429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.267507 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.267509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.268460 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.268461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.269814 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.269818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.270563 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.270564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.270648 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.270651 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.271028 (+ 377us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.271080 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":59,"thread_start_us":96,"threads_started":1}
I20260620 23:02:02.271458 29825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.267143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33011 (local address 127.21.66.65:53679)
0620 23:02:02.267393 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.267403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.267484 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.267614 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.267615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.267616 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.267689 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.267690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.268378 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.268379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.269814 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.269818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.270708 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.270708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.270793 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.270796 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.270845 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.270930 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":32,"thread_start_us":100,"threads_started":1}
I20260620 23:02:02.271461 29581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f449f188e54b1c932502499311e5bc" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519" is_pre_election: true
I20260620 23:02:02.271615 29581 raft_consensus.cc:2454] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 0.
I20260620 23:02:02.271685 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f449f188e54b1c932502499311e5bc" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a" is_pre_election: true
I20260620 23:02:02.271761 29721 raft_consensus.cc:2454] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 0.
I20260620 23:02:02.271853 29374 leader_election.cc:302] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 525252f8fbd94bf2960d2bd576251519; no voters:
I20260620 23:02:02.272053 29802 raft_consensus.cc:2781] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:02.272111 29802 raft_consensus.cc:482] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:02.272133 29802 raft_consensus.cc:3037] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.273209 29802 raft_consensus.cc:504] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.273365 29802 leader_election.cc:288] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.273623 29581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f449f188e54b1c932502499311e5bc" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519"
I20260620 23:02:02.273705 29581 raft_consensus.cc:3037] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.273672 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f449f188e54b1c932502499311e5bc" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a"
I20260620 23:02:02.273772 29721 raft_consensus.cc:3037] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.274812 29721 raft_consensus.cc:2454] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 1.
I20260620 23:02:02.274876 29581 raft_consensus.cc:2454] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 1.
I20260620 23:02:02.275067 29374 leader_election.cc:302] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 525252f8fbd94bf2960d2bd576251519; no voters:
I20260620 23:02:02.275244 29802 raft_consensus.cc:2781] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:02.275462 29802 raft_consensus.cc:686] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Running, Role: LEADER
I20260620 23:02:02.275548 29802 consensus_queue.cc:237] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.276391 29284 catalog_manager.cc:5495] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } health_report { overall_health: UNKNOWN } } }
W20260620 23:02:02.283077 29768 tablet.cc:2369] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996521, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.126@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:02:02.323788 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.315043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42301 (local address 127.0.0.1:33732)
0620 23:02:02.315244 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.315256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.315344 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.315514 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.315515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.315953 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:02.316017 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.316020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.316831 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.316833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.317699 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.317702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.318192 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.318194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.318258 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.318913 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:02.318918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:02.320605 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:02.322634 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:02.322643 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:02.322645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:02.322889 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:02.323121 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:02.323122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:02.323122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:02.323186 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:02.323481 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:02.323488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:02.323535 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.323594 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.323706 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260620 23:02:02.323904 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.315181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.21.66.126:42301)
0620 23:02:02.315235 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.315239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.315340 (+ 101us) server_negotiation.cc:407] Connection header received
0620 23:02:02.315407 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.315408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.315416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.315481 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:02.316153 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.316731 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.317815 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.318052 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.318201 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.320706 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:02.320707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:02.320708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:02.320716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:02.322421 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.322972 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.322973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.322974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.323044 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.323253 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.323254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.323255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.323407 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:02.323487 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.323641 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.323773 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":29}
I20260620 23:02:02.326431 29284 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:33732:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:02:02.326700 29284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:02.328510 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 4f03b505ca32435d84e473d8aa377dd3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63e3dca306c94f18a81e598ba8460f7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:02:02.328576 29558 tablet_service.cc:1467] Processing CreateTablet for tablet 4f03b505ca32435d84e473d8aa377dd3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63e3dca306c94f18a81e598ba8460f7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:02:02.328629 29699 tablet_service.cc:1467] Processing CreateTablet for tablet 4f03b505ca32435d84e473d8aa377dd3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63e3dca306c94f18a81e598ba8460f7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:02:02.328706 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f03b505ca32435d84e473d8aa377dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.328742 29558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f03b505ca32435d84e473d8aa377dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.328764 29699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f03b505ca32435d84e473d8aa377dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.330464 29797 tablet_bootstrap.cc:492] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Bootstrap starting.
I20260620 23:02:02.330737 29795 tablet_bootstrap.cc:492] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: Bootstrap starting.
I20260620 23:02:02.330968 29797 tablet_bootstrap.cc:654] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.331586 29795 tablet_bootstrap.cc:654] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.331636 29798 tablet_bootstrap.cc:492] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: Bootstrap starting.
I20260620 23:02:02.331789 29797 tablet_bootstrap.cc:492] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: No bootstrap required, opened a new log
I20260620 23:02:02.331854 29797 ts_tablet_manager.cc:1397] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:02.332041 29798 tablet_bootstrap.cc:654] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.332024 29797 raft_consensus.cc:348] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.332134 29797 raft_consensus.cc:374] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.332175 29797 raft_consensus.cc:729] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.332242 29797 consensus_queue.cc:260] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.332368 29797 ts_tablet_manager.cc:1428] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 23:02:02.332401 29795 tablet_bootstrap.cc:492] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: No bootstrap required, opened a new log
I20260620 23:02:02.332463 29795 ts_tablet_manager.cc:1397] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:02:02.332533 29802 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.332612 29802 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:02:02.332624 29795 raft_consensus.cc:348] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.332691 29795 raft_consensus.cc:374] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.332712 29795 raft_consensus.cc:729] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b79e6fd8d547d49e71f768c7ade66a, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.332755 29795 consensus_queue.cc:260] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.332860 29795 ts_tablet_manager.cc:1428] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:02.333027 29801 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.333112 29801 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:02:02.333525 29798 tablet_bootstrap.cc:492] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: No bootstrap required, opened a new log
I20260620 23:02:02.333617 29798 ts_tablet_manager.cc:1397] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:02:02.333791 29798 raft_consensus.cc:348] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.333920 29798 raft_consensus.cc:374] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.333972 29798 raft_consensus.cc:729] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525252f8fbd94bf2960d2bd576251519, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.334030 29798 consensus_queue.cc:260] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.334167 29798 ts_tablet_manager.cc:1428] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:02.334203 29813 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.334286 29813 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:02:02.444456 29801 raft_consensus.cc:482] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:02.444648 29801 raft_consensus.cc:504] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.445083 29801 leader_election.cc:288] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291), 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953)
I20260620 23:02:02.448067 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.445401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:38495 (local address 127.21.66.65:37291)
0620 23:02:02.445457 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.445459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.445467 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:02.445501 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.445502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.445511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.445587 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.445748 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.446355 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.447334 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.447701 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.447788 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.447797 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.447957 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.448004 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:02:02.448068 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.445276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37291 (local address 127.21.66.67:38495)
0620 23:02:02.445334 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.445351 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.445403 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.445593 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.445594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.445595 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.445672 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.445674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.446446 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.446447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.447235 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.447237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.447788 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.447789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.447872 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.447875 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.447939 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.447984 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":3}
I20260620 23:02:02.448410 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03b505ca32435d84e473d8aa377dd3" candidate_uuid: "40b79e6fd8d547d49e71f768c7ade66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" is_pre_election: true
I20260620 23:02:02.448498 29441 raft_consensus.cc:2454] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40b79e6fd8d547d49e71f768c7ade66a in term 0.
I20260620 23:02:02.448730 29652 leader_election.cc:302] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 40b79e6fd8d547d49e71f768c7ade66a; no voters:
I20260620 23:02:02.448899 29801 raft_consensus.cc:2781] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:02.448976 29801 raft_consensus.cc:482] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:02.449018 29801 raft_consensus.cc:3037] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.449190 29837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.445333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:36953 (local address 127.21.66.67:43763)
0620 23:02:02.445494 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.445505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.445601 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.445811 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.445812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.445814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:02.445931 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.445933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.446852 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.446853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.448057 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.448059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.448926 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.448927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.449004 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.449009 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.449072 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.449123 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":14,"thread_start_us":85,"threads_started":1}
I20260620 23:02:02.449230 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.445395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:43763 (local address 127.21.66.66:36953)
0620 23:02:02.445491 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.445494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.445584 (+ 90us) server_negotiation.cc:407] Connection header received
0620 23:02:02.445699 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.445700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.445711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.445807 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:02.446111 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.446751 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.448220 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.448801 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.448886 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.448897 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.449091 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.449142 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:02:02.450026 29801 raft_consensus.cc:504] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.450243 29801 leader_election.cc:288] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [CANDIDATE]: Term 1 election: Requested vote from peers 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291), 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953)
I20260620 23:02:02.450349 29581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03b505ca32435d84e473d8aa377dd3" candidate_uuid: "40b79e6fd8d547d49e71f768c7ade66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519" is_pre_election: true
I20260620 23:02:02.450387 29580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03b505ca32435d84e473d8aa377dd3" candidate_uuid: "40b79e6fd8d547d49e71f768c7ade66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519"
I20260620 23:02:02.450440 29581 raft_consensus.cc:2454] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40b79e6fd8d547d49e71f768c7ade66a in term 0.
I20260620 23:02:02.450665 29441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03b505ca32435d84e473d8aa377dd3" candidate_uuid: "40b79e6fd8d547d49e71f768c7ade66a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2"
I20260620 23:02:02.450764 29441 raft_consensus.cc:3037] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.451407 29441 raft_consensus.cc:2454] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40b79e6fd8d547d49e71f768c7ade66a in term 1.
I20260620 23:02:02.451594 29652 leader_election.cc:302] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 40b79e6fd8d547d49e71f768c7ade66a; no voters: 525252f8fbd94bf2960d2bd576251519
I20260620 23:02:02.451726 29801 raft_consensus.cc:2781] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:02.451925 29801 raft_consensus.cc:686] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [term 1 LEADER]: Becoming Leader. State: Replica: 40b79e6fd8d547d49e71f768c7ade66a, State: Running, Role: LEADER
I20260620 23:02:02.451994 29801 consensus_queue.cc:237] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.452404 29838 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: TxnStatusTablet state changed. Reason: New leader 40b79e6fd8d547d49e71f768c7ade66a. Latest consensus state: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.452521 29838 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:02:02.452749 29840 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:02:02.452888 29840 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:02:02.452785 29284 catalog_manager.cc:5495] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a reported cstate change: term changed from 0 to 1, leader changed from <none> to 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67). New cstate: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } health_report { overall_health: HEALTHY } } }
I20260620 23:02:02.483933 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.481422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.21.66.67:33011)
0620 23:02:02.481505 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.481507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.481517 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:02:02.481558 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.481558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.481578 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.481636 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:02.481774 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.482340 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.483102 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.483304 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.483384 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.483577 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.483688 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.483777 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.483825 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.483846 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:02:02.483933 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.481329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33011 (local address 127.0.0.1:52352)
0620 23:02:02.481399 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.481418 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.481485 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.481637 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.481639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.481640 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:02:02.481695 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.481698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.482423 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.482424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.483008 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.483010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.483383 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.483384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.483459 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.483477 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.483761 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.483770 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.483809 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.483862 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 23:02:02.485244 29441 raft_consensus.cc:1261] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Refusing update from remote peer 40b79e6fd8d547d49e71f768c7ade66a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.485263 29580 raft_consensus.cc:3037] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.485476 29838 consensus_queue.cc:1035] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.486042 29802 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: TxnStatusTablet state changed. Reason: New leader 40b79e6fd8d547d49e71f768c7ade66a. Latest consensus state: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.486173 29802 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:02:02.486383 29580 raft_consensus.cc:1261] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Refusing update from remote peer 40b79e6fd8d547d49e71f768c7ade66a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.486712 29838 consensus_queue.cc:1035] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.487015 29801 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.487109 29801 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:02:02.487187 29809 mvcc.cc:205] Tried to move back new op lower bound from 7299057756096794624 to 7299057755963764736. Current Snapshot: MvccSnapshot[applied={T|T < 7299057756096794624}]
I20260620 23:02:02.487475 29805 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.487550 29808 mvcc.cc:205] Tried to move back new op lower bound from 7299057756096794624 to 7299057755963764736. Current Snapshot: MvccSnapshot[applied={T|T < 7299057756096794624}]
I20260620 23:02:02.487573 29805 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:02:02.487793 29810 mvcc.cc:205] Tried to move back new op lower bound from 7299057756096794624 to 7299057755963764736. Current Snapshot: MvccSnapshot[applied={T|T < 7299057756096794624}]
I20260620 23:02:02.487869 29802 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.487947 29802 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 3619a9cc83424d16b1ddfdd47bf8f1c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:02:02.489038 29813 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: TxnStatusTablet state changed. Reason: New leader 40b79e6fd8d547d49e71f768c7ade66a. Latest consensus state: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.489161 29813 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 525252f8fbd94bf2960d2bd576251519: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:02:02.492789 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.489983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.21.66.65:37291)
0620 23:02:02.490063 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.490065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.490073 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:02.490136 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.490137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.490161 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.490239 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:02.490345 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.491096 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.491845 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.492179 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.492244 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.492409 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.492545 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.492621 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.492658 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.492680 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:02:02.499147 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.495867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:57455 (local address 127.21.66.67:33011)
0620 23:02:02.495978 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.495980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.496023 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:02:02.496119 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.496120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.496132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.496239 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:02.496465 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.496985 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.498186 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.498562 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.498621 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.498843 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.498881 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.498936 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.499061 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.499082 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:02:02.499162 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.495734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33011 (local address 127.21.66.65:57455)
0620 23:02:02.495965 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.495979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.496040 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.496245 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.496246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.496250 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:02:02.496344 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.496346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.497065 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.497066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.498019 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.498021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.498637 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.498638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.498711 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.498733 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.498945 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.498947 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.498995 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.499043 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260620 23:02:02.500061 29838 tablet_replica.cc:439] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40b79e6fd8d547d49e71f768c7ade66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } } }
I20260620 23:02:02.500171 29838 tablet_replica.cc:442] T 4f03b505ca32435d84e473d8aa377dd3 P 40b79e6fd8d547d49e71f768c7ade66a: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:02:02.501921 29721 raft_consensus.cc:1261] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Refusing update from remote peer 3619a9cc83424d16b1ddfdd47bf8f1c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.501935 29580 raft_consensus.cc:1261] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Refusing update from remote peer 3619a9cc83424d16b1ddfdd47bf8f1c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.502218 29826 consensus_queue.cc:1035] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.502372 29802 consensus_queue.cc:1035] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.511361 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.508449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:50825 (local address 127.21.66.65:37291)
0620 23:02:02.508522 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.508524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.508530 (+ 6us) server_negotiation.cc:407] Connection header received
0620 23:02:02.508551 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.508552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.508561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.508612 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:02.508780 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.509499 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.510445 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.510762 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.510826 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.511058 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.511096 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.511163 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.511255 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.511277 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:02:02.511361 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.508286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37291 (local address 127.21.66.67:50825)
0620 23:02:02.508363 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:02.508376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:02.508422 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:02.508617 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:02.508618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:02.508619 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:02:02.508683 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.508685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.509597 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.509598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.510355 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:02.510357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.510861 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.510862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:02.510939 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.510959 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.511166 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.511168 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:02:02.511228 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:02:02.511278 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 23:02:02.521400 29283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:02.521518 29283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:02.522226 29283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Sending DeleteTablet for 3 replicas of tablet 20f449f188e54b1c932502499311e5bc
I20260620 23:02:02.522470 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f449f188e54b1c932502499311e5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:43178
I20260620 23:02:02.522569 29818 tablet_replica.cc:330] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: stopping tablet replica
I20260620 23:02:02.522580 29558 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f449f188e54b1c932502499311e5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:60998
I20260620 23:02:02.522614 29818 raft_consensus.cc:2229] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:02.522701 29818 raft_consensus.cc:2258] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.522713 29699 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f449f188e54b1c932502499311e5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52348
I20260620 23:02:02.522771 29820 tablet_replica.cc:330] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: stopping tablet replica
I20260620 23:02:02.522828 29820 raft_consensus.cc:2229] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:02.522867 29820 raft_consensus.cc:2258] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.522862 29819 tablet_replica.cc:330] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: stopping tablet replica
I20260620 23:02:02.522912 29819 raft_consensus.cc:2229] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:02.522945 29819 raft_consensus.cc:2258] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.523259 29819 ts_tablet_manager.cc:1905] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.523404 29818 ts_tablet_manager.cc:1905] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.525048 29819 ts_tablet_manager.cc:1918] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:02:02.525111 29819 log.cc:1196] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/20f449f188e54b1c932502499311e5bc
I20260620 23:02:02.525152 29820 ts_tablet_manager.cc:1905] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.525332 29819 ts_tablet_manager.cc:1939] T 20f449f188e54b1c932502499311e5bc P 40b79e6fd8d547d49e71f768c7ade66a: Deleting consensus metadata
I20260620 23:02:02.525514 29818 ts_tablet_manager.cc:1918] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:02:02.525559 29818 log.cc:1196] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/20f449f188e54b1c932502499311e5bc
I20260620 23:02:02.525741 29818 ts_tablet_manager.cc:1939] T 20f449f188e54b1c932502499311e5bc P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting consensus metadata
I20260620 23:02:02.526057 29268 catalog_manager.cc:4841] TS 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291): tablet 20f449f188e54b1c932502499311e5bc (table test-table [id=34c706d9ce204da1b2353dc169c190d3]) successfully deleted
I20260620 23:02:02.527041 29820 ts_tablet_manager.cc:1918] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:02:02.527100 29820 log.cc:1196] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/20f449f188e54b1c932502499311e5bc
I20260620 23:02:02.527318 29820 ts_tablet_manager.cc:1939] T 20f449f188e54b1c932502499311e5bc P 525252f8fbd94bf2960d2bd576251519: Deleting consensus metadata
I20260620 23:02:02.527534 29270 catalog_manager.cc:4841] TS 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011): tablet 20f449f188e54b1c932502499311e5bc (table test-table [id=34c706d9ce204da1b2353dc169c190d3]) successfully deleted
I20260620 23:02:02.527885 29270 catalog_manager.cc:4841] TS 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953): tablet 20f449f188e54b1c932502499311e5bc (table test-table [id=34c706d9ce204da1b2353dc169c190d3]) successfully deleted
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:02:02.548920 29333 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:02.543340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.21.66.126:42301)
0620 23:02:02.543393 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.543395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.543632 (+ 237us) server_negotiation.cc:407] Connection header received
0620 23:02:02.543715 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.543716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.543727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.543795 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:02.544407 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.544919 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.545694 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.545898 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.545984 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.548868 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33738: BlockingRecv error: recv got EOF from 127.0.0.1:33738 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 23:02:02 dist-test-slave-qgtk krb5kdc[29239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM
I20260620 23:02:02.557510 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.550085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.21.66.126:42301)
0620 23:02:02.550159 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.550161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.550469 (+ 308us) server_negotiation.cc:407] Connection header received
0620 23:02:02.550556 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.550557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.550568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.550623 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:02.551090 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.551579 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.552364 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.552571 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.552647 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.554922 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:02.554923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:02.554923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:02.554932 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:02.556340 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.556710 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.556711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.556711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.556771 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:02.556976 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:02.556977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:02.556977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:02.557093 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:02.557204 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.557312 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.557391 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 23:02:02.559376 29283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:02:02.559580 29283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:02.560881 29421 tablet_service.cc:1467] Processing CreateTablet for tablet 01286db78e414ec2bef70e5b3c979f10 (DEFAULT_TABLE table=test-table [id=3dbcaf7697be4071a570a88b872dd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.560949 29558 tablet_service.cc:1467] Processing CreateTablet for tablet 01286db78e414ec2bef70e5b3c979f10 (DEFAULT_TABLE table=test-table [id=3dbcaf7697be4071a570a88b872dd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.561038 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01286db78e414ec2bef70e5b3c979f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.561057 29558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01286db78e414ec2bef70e5b3c979f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.562315 29699 tablet_service.cc:1467] Processing CreateTablet for tablet 01286db78e414ec2bef70e5b3c979f10 (DEFAULT_TABLE table=test-table [id=3dbcaf7697be4071a570a88b872dd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.562428 29699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01286db78e414ec2bef70e5b3c979f10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.562439 29797 tablet_bootstrap.cc:492] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Bootstrap starting.
I20260620 23:02:02.562848 29797 tablet_bootstrap.cc:654] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.563861 29795 tablet_bootstrap.cc:492] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: Bootstrap starting.
I20260620 23:02:02.564325 29795 tablet_bootstrap.cc:654] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.564391 29798 tablet_bootstrap.cc:492] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: Bootstrap starting.
I20260620 23:02:02.564764 29798 tablet_bootstrap.cc:654] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.565312 29798 tablet_bootstrap.cc:492] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: No bootstrap required, opened a new log
I20260620 23:02:02.565363 29798 ts_tablet_manager.cc:1397] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:02.565418 29797 tablet_bootstrap.cc:492] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: No bootstrap required, opened a new log
I20260620 23:02:02.565457 29797 ts_tablet_manager.cc:1397] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 23:02:02.565493 29798 raft_consensus.cc:348] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.565573 29798 raft_consensus.cc:374] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.565591 29798 raft_consensus.cc:729] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525252f8fbd94bf2960d2bd576251519, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.565631 29798 consensus_queue.cc:260] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.565613 29797 raft_consensus.cc:348] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.565671 29797 raft_consensus.cc:374] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.565691 29797 raft_consensus.cc:729] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.565728 29797 consensus_queue.cc:260] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.565761 29798 ts_tablet_manager.cc:1428] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:02.565822 29797 ts_tablet_manager.cc:1428] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:02.566494 29795 tablet_bootstrap.cc:492] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: No bootstrap required, opened a new log
I20260620 23:02:02.566545 29795 ts_tablet_manager.cc:1397] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260620 23:02:02.566866 29795 raft_consensus.cc:348] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.566928 29795 raft_consensus.cc:374] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.566941 29795 raft_consensus.cc:729] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b79e6fd8d547d49e71f768c7ade66a, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.567013 29795 consensus_queue.cc:260] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.567114 29795 ts_tablet_manager.cc:1428] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:02:02.709879 29852 raft_consensus.cc:482] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:02.710011 29852 raft_consensus.cc:504] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.710237 29852 leader_election.cc:288] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.710548 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01286db78e414ec2bef70e5b3c979f10" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a" is_pre_election: true
I20260620 23:02:02.710589 29580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01286db78e414ec2bef70e5b3c979f10" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519" is_pre_election: true
I20260620 23:02:02.710664 29721 raft_consensus.cc:2454] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 0.
I20260620 23:02:02.710671 29580 raft_consensus.cc:2454] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 0.
I20260620 23:02:02.710834 29374 leader_election.cc:302] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 40b79e6fd8d547d49e71f768c7ade66a; no voters:
I20260620 23:02:02.710989 29852 raft_consensus.cc:2781] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:02.711061 29852 raft_consensus.cc:482] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:02.711092 29852 raft_consensus.cc:3037] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.711833 29852 raft_consensus.cc:504] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.711979 29852 leader_election.cc:288] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.712191 29580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01286db78e414ec2bef70e5b3c979f10" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519"
I20260620 23:02:02.712225 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01286db78e414ec2bef70e5b3c979f10" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a"
I20260620 23:02:02.712296 29580 raft_consensus.cc:3037] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.712292 29721 raft_consensus.cc:3037] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.712922 29721 raft_consensus.cc:2454] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 1.
I20260620 23:02:02.713053 29580 raft_consensus.cc:2454] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 1.
I20260620 23:02:02.713094 29374 leader_election.cc:302] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 40b79e6fd8d547d49e71f768c7ade66a; no voters:
I20260620 23:02:02.713179 29852 raft_consensus.cc:2781] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:02.713230 29852 raft_consensus.cc:686] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Running, Role: LEADER
I20260620 23:02:02.713307 29852 consensus_queue.cc:237] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.714018 29284 catalog_manager.cc:5495] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:02.751240 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:02.748656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.21.66.65:37291)
0620 23:02:02.748730 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:02.748733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:02.748770 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:02:02.748876 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:02.748878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:02.748888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:02.748945 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:02.749133 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.749610 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.750537 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:02.750730 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:02.750805 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:02.750952 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:02.750989 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:02.751051 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:02.751114 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 23:02:02.751135 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:02:02.752139 29721 raft_consensus.cc:1261] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Refusing update from remote peer 3619a9cc83424d16b1ddfdd47bf8f1c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.752170 29580 raft_consensus.cc:1261] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Refusing update from remote peer 3619a9cc83424d16b1ddfdd47bf8f1c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.752410 29826 consensus_queue.cc:1035] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.752523 29852 consensus_queue.cc:1035] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.755168 29283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:02.755263 29283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:02.755975 29283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Sending DeleteTablet for 3 replicas of tablet 01286db78e414ec2bef70e5b3c979f10
I20260620 23:02:02.756202 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet 01286db78e414ec2bef70e5b3c979f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:43178
I20260620 23:02:02.756306 29818 tablet_replica.cc:330] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: stopping tablet replica
I20260620 23:02:02.756309 29699 tablet_service.cc:1514] Processing DeleteTablet for tablet 01286db78e414ec2bef70e5b3c979f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52348
I20260620 23:02:02.756366 29818 raft_consensus.cc:2229] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:02.756397 29558 tablet_service.cc:1514] Processing DeleteTablet for tablet 01286db78e414ec2bef70e5b3c979f10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:02 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:60998
I20260620 23:02:02.756424 29818 raft_consensus.cc:2258] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.756477 29819 tablet_replica.cc:330] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: stopping tablet replica
I20260620 23:02:02.756424 29283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:02:02.756542 29819 raft_consensus.cc:2229] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 23:02:02.756562 29283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:02.756592 29819 raft_consensus.cc:2258] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.756666 29820 tablet_replica.cc:330] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: stopping tablet replica
I20260620 23:02:02.756726 29820 raft_consensus.cc:2229] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:02.756763 29820 raft_consensus.cc:2258] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:02.757153 29820 ts_tablet_manager.cc:1905] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.757879 29699 tablet_service.cc:1467] Processing CreateTablet for tablet abf87785692a46f0be12488c17e7816a (DEFAULT_TABLE table=test-table [id=3f8e759fb13144179ed209d1669b031a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.758006 29699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf87785692a46f0be12488c17e7816a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.758898 29820 ts_tablet_manager.cc:1918] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:02.758967 29820 log.cc:1196] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/01286db78e414ec2bef70e5b3c979f10
I20260620 23:02:02.758972 29421 tablet_service.cc:1467] Processing CreateTablet for tablet abf87785692a46f0be12488c17e7816a (DEFAULT_TABLE table=test-table [id=3f8e759fb13144179ed209d1669b031a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.759111 29421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf87785692a46f0be12488c17e7816a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.759223 29820 ts_tablet_manager.cc:1939] T 01286db78e414ec2bef70e5b3c979f10 P 525252f8fbd94bf2960d2bd576251519: Deleting consensus metadata
I20260620 23:02:02.759457 29795 tablet_bootstrap.cc:492] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: Bootstrap starting.
I20260620 23:02:02.759562 29270 catalog_manager.cc:4841] TS 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953): tablet 01286db78e414ec2bef70e5b3c979f10 (table test-table [id=3dbcaf7697be4071a570a88b872dd0e2]) successfully deleted
I20260620 23:02:02.759778 29795 tablet_bootstrap.cc:654] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.760417 29795 tablet_bootstrap.cc:492] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: No bootstrap required, opened a new log
I20260620 23:02:02.760471 29795 ts_tablet_manager.cc:1397] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:02:02.758738 29558 tablet_service.cc:1467] Processing CreateTablet for tablet abf87785692a46f0be12488c17e7816a (DEFAULT_TABLE table=test-table [id=3f8e759fb13144179ed209d1669b031a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:02.760591 29558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf87785692a46f0be12488c17e7816a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:02.760598 29797 tablet_bootstrap.cc:492] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Bootstrap starting.
I20260620 23:02:02.760594 29795 raft_consensus.cc:348] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.760666 29795 raft_consensus.cc:374] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.760699 29795 raft_consensus.cc:729] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b79e6fd8d547d49e71f768c7ade66a, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.760749 29795 consensus_queue.cc:260] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.760843 29795 ts_tablet_manager.cc:1428] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:02.760910 29797 tablet_bootstrap.cc:654] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.761570 29797 tablet_bootstrap.cc:492] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2: No bootstrap required, opened a new log
I20260620 23:02:02.761624 29797 ts_tablet_manager.cc:1397] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:02.761682 29798 tablet_bootstrap.cc:492] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: Bootstrap starting.
I20260620 23:02:02.761747 29797 raft_consensus.cc:348] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.761847 29797 raft_consensus.cc:374] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.761876 29797 raft_consensus.cc:729] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.761914 29818 ts_tablet_manager.cc:1905] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.761981 29798 tablet_bootstrap.cc:654] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:02.762594 29798 tablet_bootstrap.cc:492] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: No bootstrap required, opened a new log
I20260620 23:02:02.762650 29798 ts_tablet_manager.cc:1397] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:02.762775 29798 raft_consensus.cc:348] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.762877 29798 raft_consensus.cc:374] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:02.762900 29798 raft_consensus.cc:729] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525252f8fbd94bf2960d2bd576251519, State: Initialized, Role: FOLLOWER
I20260620 23:02:02.762941 29798 consensus_queue.cc:260] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.763038 29798 ts_tablet_manager.cc:1428] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 23:02:02.763372 29818 ts_tablet_manager.cc:1918] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:02.763419 29818 log.cc:1196] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/01286db78e414ec2bef70e5b3c979f10
I20260620 23:02:02.763578 29818 ts_tablet_manager.cc:1939] T 01286db78e414ec2bef70e5b3c979f10 P 3619a9cc83424d16b1ddfdd47bf8f1c2: Deleting consensus metadata
I20260620 23:02:02.764261 29268 catalog_manager.cc:4841] TS 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291): tablet 01286db78e414ec2bef70e5b3c979f10 (table test-table [id=3dbcaf7697be4071a570a88b872dd0e2]) successfully deleted
I20260620 23:02:02.761924 29797 consensus_queue.cc:260] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.764394 29797 ts_tablet_manager.cc:1428] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 23:02:02.764745 29819 ts_tablet_manager.cc:1905] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:02.766109 29819 ts_tablet_manager.cc:1918] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:02.766217 29819 log.cc:1196] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/01286db78e414ec2bef70e5b3c979f10
I20260620 23:02:02.766418 29819 ts_tablet_manager.cc:1939] T 01286db78e414ec2bef70e5b3c979f10 P 40b79e6fd8d547d49e71f768c7ade66a: Deleting consensus metadata
I20260620 23:02:02.766814 29270 catalog_manager.cc:4841] TS 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011): tablet 01286db78e414ec2bef70e5b3c979f10 (table test-table [id=3dbcaf7697be4071a570a88b872dd0e2]) successfully deleted
I20260620 23:02:02.945799 29852 raft_consensus.cc:482] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:02.945909 29852 raft_consensus.cc:504] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.946166 29852 leader_election.cc:288] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.946508 29580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf87785692a46f0be12488c17e7816a" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519" is_pre_election: true
I20260620 23:02:02.946621 29580 raft_consensus.cc:2454] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 0.
I20260620 23:02:02.946571 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf87785692a46f0be12488c17e7816a" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a" is_pre_election: true
I20260620 23:02:02.946710 29721 raft_consensus.cc:2454] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 0.
I20260620 23:02:02.946823 29374 leader_election.cc:302] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 525252f8fbd94bf2960d2bd576251519; no voters:
I20260620 23:02:02.946918 29852 raft_consensus.cc:2781] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:02.946962 29852 raft_consensus.cc:482] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:02.946985 29852 raft_consensus.cc:3037] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.947714 29852 raft_consensus.cc:504] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.947865 29852 leader_election.cc:288] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953), 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011)
I20260620 23:02:02.948031 29580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf87785692a46f0be12488c17e7816a" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525252f8fbd94bf2960d2bd576251519"
I20260620 23:02:02.948074 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf87785692a46f0be12488c17e7816a" candidate_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b79e6fd8d547d49e71f768c7ade66a"
I20260620 23:02:02.948125 29580 raft_consensus.cc:3037] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.948148 29721 raft_consensus.cc:3037] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:02.948953 29721 raft_consensus.cc:2454] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 1.
I20260620 23:02:02.948979 29580 raft_consensus.cc:2454] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3619a9cc83424d16b1ddfdd47bf8f1c2 in term 1.
I20260620 23:02:02.949144 29374 leader_election.cc:302] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3619a9cc83424d16b1ddfdd47bf8f1c2, 525252f8fbd94bf2960d2bd576251519; no voters:
I20260620 23:02:02.949259 29852 raft_consensus.cc:2781] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:02.949337 29852 raft_consensus.cc:686] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3619a9cc83424d16b1ddfdd47bf8f1c2, State: Running, Role: LEADER
I20260620 23:02:02.949395 29852 consensus_queue.cc:237] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } }
I20260620 23:02:02.950081 29283 catalog_manager.cc:5495] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3619a9cc83424d16b1ddfdd47bf8f1c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:02.992399 29721 raft_consensus.cc:1261] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Refusing update from remote peer 3619a9cc83424d16b1ddfdd47bf8f1c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.992436 29580 raft_consensus.cc:1261] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Refusing update from remote peer 3619a9cc83424d16b1ddfdd47bf8f1c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:02.992686 29852 consensus_queue.cc:1035] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525252f8fbd94bf2960d2bd576251519" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:02.992825 29826 consensus_queue.cc:1035] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b79e6fd8d547d49e71f768c7ade66a" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:03.003157 29283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:03.003293 29283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:03.003832 29283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6372c8dfabc4d40b19a4726e76d5b54: Sending DeleteTablet for 3 replicas of tablet abf87785692a46f0be12488c17e7816a
I20260620 23:02:03.004117 29699 tablet_service.cc:1514] Processing DeleteTablet for tablet abf87785692a46f0be12488c17e7816a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:03 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52348
I20260620 23:02:03.004137 29421 tablet_service.cc:1514] Processing DeleteTablet for tablet abf87785692a46f0be12488c17e7816a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:03 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:43178
I20260620 23:02:03.004165 29558 tablet_service.cc:1514] Processing DeleteTablet for tablet abf87785692a46f0be12488c17e7816a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:03 UTC) from {username='oryx', principal='oryx/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:60998
I20260620 23:02:03.004230 29818 tablet_replica.cc:330] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2: stopping tablet replica
I20260620 23:02:03.004244 29819 tablet_replica.cc:330] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: stopping tablet replica
I20260620 23:02:03.004254 29820 tablet_replica.cc:330] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: stopping tablet replica
I20260620 23:02:03.004280 29818 raft_consensus.cc:2229] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:02:03.004299 29819 raft_consensus.cc:2229] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:03.004304 29820 raft_consensus.cc:2229] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:03.004330 29818 raft_consensus.cc:2258] T abf87785692a46f0be12488c17e7816a P 3619a9cc83424d16b1ddfdd47bf8f1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:03.004334 29819 raft_consensus.cc:2258] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:03.004338 29820 raft_consensus.cc:2258] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:03.004681 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 29356
I20260620 23:02:03.004768 29820 ts_tablet_manager.cc:1905] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:03.006323 29819 ts_tablet_manager.cc:1905] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:03.006553 29820 ts_tablet_manager.cc:1918] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:02:03.006618 29820 log.cc:1196] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/abf87785692a46f0be12488c17e7816a
I20260620 23:02:03.006824 29820 ts_tablet_manager.cc:1939] T abf87785692a46f0be12488c17e7816a P 525252f8fbd94bf2960d2bd576251519: Deleting consensus metadata
I20260620 23:02:03.007158 29270 catalog_manager.cc:4841] TS 525252f8fbd94bf2960d2bd576251519 (127.21.66.66:36953): tablet abf87785692a46f0be12488c17e7816a (table test-table [id=3f8e759fb13144179ed209d1669b031a]) successfully deleted
I20260620 23:02:03.007920 29819 ts_tablet_manager.cc:1918] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:02:03.007997 29819 log.cc:1196] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/abf87785692a46f0be12488c17e7816a
I20260620 23:02:03.008214 29819 ts_tablet_manager.cc:1939] T abf87785692a46f0be12488c17e7816a P 40b79e6fd8d547d49e71f768c7ade66a: Deleting consensus metadata
I20260620 23:02:03.008577 29270 catalog_manager.cc:4841] TS 40b79e6fd8d547d49e71f768c7ade66a (127.21.66.67:33011): tablet abf87785692a46f0be12488c17e7816a (table test-table [id=3f8e759fb13144179ed209d1669b031a]) successfully deleted
W20260620 23:02:03.011806 29268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:37291 (error 108)
I20260620 23:02:03.011904 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 29496
I20260620 23:02:03.013034 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:03.012767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37291 (local address 127.0.0.1:43216)
0620 23:02:03.012879 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:03.012981 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
W20260620 23:02:03.013164 29268 catalog_manager.cc:4584] TS 3619a9cc83424d16b1ddfdd47bf8f1c2 (127.21.66.65:37291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abf87785692a46f0be12488c17e7816a: Network error: Client connection negotiation failed: client connection to 127.21.66.65:37291: connect: Connection refused (error 111)
I20260620 23:02:03.018385 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 29636
I20260620 23:02:03.024633 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 29252
2026-06-20T23:02:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2883 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:02:03 dist-test-slave-qgtk krb5kdc[29884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:02:03 dist-test-slave-qgtk krb5kdc[29884](info): set up 2 sockets
Jun 20 23:02:03 dist-test-slave-qgtk krb5kdc[29884](info): commencing operation
krb5kdc: starting...
W20260620 23:02:04.151279 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:02:04 dist-test-slave-qgtk krb5kdc[29884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:02:04.273820 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:33861
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:04.360455 29897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:04.360724 29897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:04.360766 29897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:04.361976 29897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:04.362046 29897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:04.362068 29897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:04.362088 29897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:04.362152 29897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:04.363665 29897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:04.363973 29897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:04.364204 29897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:04.370048 29902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:04.370074 29905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:04.370048 29903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:04.370076 29897 server_base.cc:1034] running on GCE node
I20260620 23:02:04.370574 29897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:04.370834 29897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:04.371976 29897 hybrid_clock.cc:648] HybridClock initialized: now 1781996524371960 us; error 43 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T23:02:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1356 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:02:04 dist-test-slave-qgtk krb5kdc[29910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:02:04 dist-test-slave-qgtk krb5kdc[29910](info): set up 2 sockets
Jun 20 23:02:04 dist-test-slave-qgtk krb5kdc[29910](info): commencing operation
krb5kdc: starting...
W20260620 23:02:05.467598 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:02:05 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:02:05.593562 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46345
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:05.684733 29923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:05.685019 29923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:05.685065 29923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:05.686579 29923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:05.686654 29923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:05.686684 29923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:05.686720 29923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:05.686762 29923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:05.688482 29923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:05.688800 29923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:05.689052 29923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:05.694582 29931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:05.694589 29928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:05.694621 29929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:05.694623 29923 server_base.cc:1034] running on GCE node
I20260620 23:02:05.695104 29923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:05.695459 29923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:05.696627 29923 hybrid_clock.cc:648] HybridClock initialized: now 1781996525696562 us; error 83 us; skew 500 ppm
Jun 20 23:02:05 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996525, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:05.703022 29923 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:02:05.703567 29923 webserver.cc:460] Webserver started at http://127.21.66.126:45489/ using document root <none> and password file <none>
I20260620 23:02:05.703840 29923 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:05.703923 29923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:05.704134 29923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:05.705377 29923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "82a077732a464f1585808dfa06901cbb"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "a3933949b08bad68b414172817a7f1cf"
server_key_iv: "1439165567855ab62c65e3056074a06f"
server_key_version: "encryptionkey@0"
I20260620 23:02:05.705823 29923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "82a077732a464f1585808dfa06901cbb"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "a3933949b08bad68b414172817a7f1cf"
server_key_iv: "1439165567855ab62c65e3056074a06f"
server_key_version: "encryptionkey@0"
I20260620 23:02:05.707438 29923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 23:02:05.708307 29938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:05.708494 29923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:05.708616 29923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "82a077732a464f1585808dfa06901cbb"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "a3933949b08bad68b414172817a7f1cf"
server_key_iv: "1439165567855ab62c65e3056074a06f"
server_key_version: "encryptionkey@0"
I20260620 23:02:05.708724 29923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:05.728518 29923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:05.731410 29923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:05.731614 29923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:05.736245 29923 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:46345
I20260620 23:02:05.736275 29990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:46345 every 8 connection(s)
I20260620 23:02:05.736699 29923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:02:05.737644 29991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:05.739367 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 29923
I20260620 23:02:05.739492 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:02:05.739686 21769 external_mini_cluster.cc:1428] Setting key 89b913639aa187429e3e3d023d8ddbe5
I20260620 23:02:05.740566 29991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Bootstrap starting.
I20260620 23:02:05.741168 29991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:05.741493 29991 log.cc:824] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:05.742352 29991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: No bootstrap required, opened a new log
I20260620 23:02:05.744493 29991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:05.744645 29991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:05.744668 29991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a077732a464f1585808dfa06901cbb, State: Initialized, Role: FOLLOWER
I20260620 23:02:05.744807 29991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:05.744884 29991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:05.744907 29991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:05.744971 29991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:05.745735 29991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:05.745857 29991 leader_election.cc:302] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82a077732a464f1585808dfa06901cbb; no voters:
I20260620 23:02:05.746028 29991 leader_election.cc:288] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:05.746235 29997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:05.746349 29991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:05.746410 29997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 LEADER]: Becoming Leader. State: Replica: 82a077732a464f1585808dfa06901cbb, State: Running, Role: LEADER
I20260620 23:02:05.746551 29997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:05.746883 29997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82a077732a464f1585808dfa06901cbb. Latest consensus state: current_term: 1 leader_uuid: "82a077732a464f1585808dfa06901cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } } }
I20260620 23:02:05.746891 29998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82a077732a464f1585808dfa06901cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } } }
I20260620 23:02:05.746968 29997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:05.746970 29998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: This master's current role is: LEADER
Jun 20 23:02:05 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:02:05.747803 30005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:05.748332 30005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:05.750058 30005 catalog_manager.cc:1347] Generated new cluster ID: b518eafe8c5949bf9ea9b8fa37beec27
I20260620 23:02:05.750146 30005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:05.752238 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:05.740238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.21.66.126:46345)
0620 23:02:05.740430 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:05.740439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:05.740471 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:02:05.740997 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:05.741006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:05.741132 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:05.741330 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:05.742190 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:05.743266 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:05.744526 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:05.745163 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:05.745251 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:05.748335 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:05.748340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:05.748340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:05.748354 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:05.750542 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:05.750964 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:05.750967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:05.750967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:05.751069 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:05.751290 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:05.751290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:05.751291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:05.751588 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:05.751744 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:05.751872 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 23:02:05.752016 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260620 23:02:05.763559 30005 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:05.764391 30005 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:05.771070 30005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Generated new TSK 0
I20260620 23:02:05.771332 30005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:02:05.799422 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46345
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:05.892447 30017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:05.892693 30017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:05.892721 30017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:05.894065 30017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:05.894184 30017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:05.894263 30017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:02:05.895884 30017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:05.896152 30017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:05.896412 30017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:05.901757 30022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:05.901757 30023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:05.901777 30025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:05.902072 30017 server_base.cc:1034] running on GCE node
I20260620 23:02:05.902403 30017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:05.902691 30017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:05.903908 30017 hybrid_clock.cc:648] HybridClock initialized: now 1781996525903869 us; error 41 us; skew 500 ppm
Jun 20 23:02:05 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996525, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:05.906631 30017 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:02:05.907033 30017 webserver.cc:460] Webserver started at http://127.21.66.65:44455/ using document root <none> and password file <none>
I20260620 23:02:05.907229 30017 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:05.907270 30017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:05.907358 30017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:05.908368 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "c2d24bae6b3c4cb498b402d58fcb0497"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "fe31d6a2b35f485ae5fbb5fb0769fae9"
server_key_iv: "78293c9c7bb044c2c67ff180d7f414e0"
server_key_version: "encryptionkey@0"
I20260620 23:02:05.908722 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "c2d24bae6b3c4cb498b402d58fcb0497"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "fe31d6a2b35f485ae5fbb5fb0769fae9"
server_key_iv: "78293c9c7bb044c2c67ff180d7f414e0"
server_key_version: "encryptionkey@0"
I20260620 23:02:05.910061 30017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:02:05.911211 30032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:05.911405 30017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:05.911518 30017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "c2d24bae6b3c4cb498b402d58fcb0497"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "fe31d6a2b35f485ae5fbb5fb0769fae9"
server_key_iv: "78293c9c7bb044c2c67ff180d7f414e0"
server_key_version: "encryptionkey@0"
I20260620 23:02:05.911617 30017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:05.933482 30017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:05.935869 30017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:05.936053 30017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:05.936316 30017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:05.936647 30017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:05.936681 30017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:05.936703 30017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:05.936717 30017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:05.942847 30017 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:37299
I20260620 23:02:05.942907 30145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:37299 every 8 connection(s)
I20260620 23:02:05.943261 30017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:02:05.946691 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30017
I20260620 23:02:05.946826 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:02:05.947017 21769 external_mini_cluster.cc:1428] Setting key d41bfc8899756270cfd19fd12d43d0c3
Jun 20 23:02:05 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996525, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:02:05.953935 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:05.944183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:39421 (local address 127.21.66.126:46345)
0620 23:02:05.944270 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:05.944272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:05.944386 (+ 114us) server_negotiation.cc:407] Connection header received
0620 23:02:05.944879 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:05.944881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:05.944891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:05.944946 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:05.945921 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:05.946518 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:05.948093 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:05.948430 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:05.948507 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:05.950816 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:05.950818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:05.950818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:05.950826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:05.952353 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:05.952894 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:05.952896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:05.952896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:05.952962 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:05.953235 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:05.953236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:05.953236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:05.953361 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:05.953443 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:05.953742 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:02:05.953838 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:02:05.954195 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:05.944058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:39421)
0620 23:02:05.944232 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:05.944277 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:05.944792 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:05.945001 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:05.945005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:05.945609 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:05.945838 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:05.945843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:05.946746 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:05.946747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:05.948015 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:05.948018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:05.948517 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:05.948517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:05.948628 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:05.949229 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:05.949239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:05.950709 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:05.952495 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:05.952503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:05.952506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:05.952793 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:05.953043 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:05.953044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:05.953044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:05.953164 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:05.953432 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:05.953439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:05.953600 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:02:05.953710 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:02:05.953880 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260620 23:02:05.954772 30146 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46345
I20260620 23:02:05.954864 30146 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:05.955080 30146 heartbeater.cc:511] Master 127.21.66.126:46345 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:02:05.955693 29955 ts_manager.cc:194] Registered new tserver with Master: c2d24bae6b3c4cb498b402d58fcb0497 (127.21.66.65:37299)
I20260620 23:02:05.956336 29955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:39421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:02:05.978255 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46345
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:06.064523 30151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:06.064810 30151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:06.064854 30151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:06.066366 30151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:06.066470 30151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:06.066524 30151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:02:06.068203 30151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:06.068539 30151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:06.068835 30151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:06.074734 30156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.074734 30157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.074781 30159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:06.076001 30151 server_base.cc:1034] running on GCE node
I20260620 23:02:06.076283 30151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:06.076529 30151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:06.077716 30151 hybrid_clock.cc:648] HybridClock initialized: now 1781996526077694 us; error 44 us; skew 500 ppm
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:06.080695 30151 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:02:06.081245 30151 webserver.cc:460] Webserver started at http://127.21.66.66:42885/ using document root <none> and password file <none>
I20260620 23:02:06.081507 30151 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:06.081575 30151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:06.081764 30151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:06.082931 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "748db8c8451547a9a05d2216b96f7995"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "270711a220de02cb22fc9ef6e8e3170c"
server_key_iv: "900f7211c45418230ddfd31065a29573"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.083314 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "748db8c8451547a9a05d2216b96f7995"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "270711a220de02cb22fc9ef6e8e3170c"
server_key_iv: "900f7211c45418230ddfd31065a29573"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.084769 30151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:02:06.085610 30166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.085865 30151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:06.085985 30151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "748db8c8451547a9a05d2216b96f7995"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "270711a220de02cb22fc9ef6e8e3170c"
server_key_iv: "900f7211c45418230ddfd31065a29573"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.086092 30151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:06.114049 30151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:06.116971 30151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:06.117168 30151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:06.117452 30151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:06.117801 30151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:06.117861 30151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.117908 30151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:06.117944 30151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.124367 30151 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:37125
I20260620 23:02:06.124394 30279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:37125 every 8 connection(s)
I20260620 23:02:06.124799 30151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:02:06.134701 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30151
I20260620 23:02:06.134835 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:02:06.135130 21769 external_mini_cluster.cc:1428] Setting key 0d2d3b880af428e108d6b4dcc2c93d26
I20260620 23:02:06.136497 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.125670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:54505 (local address 127.21.66.126:46345)
0620 23:02:06.125759 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.125761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.126076 (+ 315us) server_negotiation.cc:407] Connection header received
0620 23:02:06.126607 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.126609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.126621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.126690 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.128000 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.128539 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.129821 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.130014 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.130082 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.132735 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:06.132736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:06.132737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:06.132744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:06.134406 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:06.135144 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:06.135146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:06.135146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:06.135225 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:06.135641 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:06.135643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:06.135643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:06.135783 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:06.135871 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:06.136293 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 23:02:06.136391 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:02:06.136901 30282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.125658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:54505)
0620 23:02:06.125854 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.125899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.126475 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.126805 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.126810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.127640 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.127877 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.127883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.128674 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.128675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.129716 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.129718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.130279 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.130282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.130428 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.130987 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.130996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.132624 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:06.134559 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:06.134571 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:06.134574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:06.135028 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:06.135334 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:06.135336 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:06.135336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:06.135544 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:06.135879 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:06.135887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:06.136132 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 23:02:06.136260 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:02:06.136455 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260620 23:02:06.137491 30280 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46345
I20260620 23:02:06.137621 30280 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:06.137831 30280 heartbeater.cc:511] Master 127.21.66.126:46345 requested a full tablet report, sending...
I20260620 23:02:06.138341 29955 ts_manager.cc:194] Registered new tserver with Master: 748db8c8451547a9a05d2216b96f7995 (127.21.66.66:37125)
I20260620 23:02:06.138746 29955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:54505
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:02:06.166210 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46345
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:06.256004 30285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:06.256246 30285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:06.256273 30285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:06.257478 30285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:06.257529 30285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:06.257556 30285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:02:06.259150 30285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:06.259387 30285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:06.259588 30285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:06.264998 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.265102 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.264998 30293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:06.265367 30285 server_base.cc:1034] running on GCE node
I20260620 23:02:06.265635 30285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:06.265902 30285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:06.267252 30285 hybrid_clock.cc:648] HybridClock initialized: now 1781996526267228 us; error 38 us; skew 500 ppm
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:06.270438 30285 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:02:06.271026 30285 webserver.cc:460] Webserver started at http://127.21.66.67:46619/ using document root <none> and password file <none>
I20260620 23:02:06.271286 30285 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:06.271337 30285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:06.271515 30285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:06.272701 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "4a9ed630a0e046b5bb3b7aaa73926d29"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "13b12f04d09f6a84a1ef8a6a8859effd"
server_key_iv: "31221192903c47644c8148f7e67d0697"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.273087 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "4a9ed630a0e046b5bb3b7aaa73926d29"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "13b12f04d09f6a84a1ef8a6a8859effd"
server_key_iv: "31221192903c47644c8148f7e67d0697"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.274617 30285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:02:06.275542 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.275853 30285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:06.275962 30285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "4a9ed630a0e046b5bb3b7aaa73926d29"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "13b12f04d09f6a84a1ef8a6a8859effd"
server_key_iv: "31221192903c47644c8148f7e67d0697"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.276047 30285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:06.287119 30285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:06.289683 30285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:06.289852 30285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:06.290098 30285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:06.290479 30285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:06.290519 30285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.290542 30285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:06.290593 30285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.296913 30285 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:45701
I20260620 23:02:06.297147 30413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:45701 every 8 connection(s)
I20260620 23:02:06.297356 30285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:02:06.302170 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30285
I20260620 23:02:06.302291 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:02:06.302484 21769 external_mini_cluster.cc:1428] Setting key 399b052efab540ae8bc5a040a273c5d7
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:02:06.308125 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.298251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:52591 (local address 127.21.66.126:46345)
0620 23:02:06.298340 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.298342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.298577 (+ 235us) server_negotiation.cc:407] Connection header received
0620 23:02:06.299090 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.299092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.299105 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.299175 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.300307 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.300831 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.302009 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.302443 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.302528 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.305085 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:06.305086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:06.305086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:06.305095 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:06.306571 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:06.307119 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:06.307120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:06.307120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:06.307181 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:06.307459 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:06.307459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:06.307459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:06.307573 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:06.307625 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:06.307987 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 23:02:06.308055 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:02:06.308457 30416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.298275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:52591)
0620 23:02:06.298445 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.298494 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.298981 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.299245 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.299249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.299889 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.300193 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.300199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.300935 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.300936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.301900 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.301903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.302558 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.302560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.302728 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.303431 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.303442 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.304958 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:06.306670 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:06.306677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:06.306679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:06.306942 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:06.307273 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:06.307274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:06.307275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:06.307393 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:06.307670 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:06.307678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:06.307842 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 23:02:06.307949 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:02:06.308209 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260620 23:02:06.309006 30414 heartbeater.cc:347] Connected to a master server at 127.21.66.126:46345
I20260620 23:02:06.309199 30414 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:06.309415 30414 heartbeater.cc:511] Master 127.21.66.126:46345 requested a full tablet report, sending...
I20260620 23:02:06.309875 29955 ts_manager.cc:194] Registered new tserver with Master: 4a9ed630a0e046b5bb3b7aaa73926d29 (127.21.66.67:45701)
I20260620 23:02:06.310343 29955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:52591
I20260620 23:02:06.314455 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab.
Entry for principal oryx/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab.
WARNING: no policy specified for oryx/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab.
Entry for principal oryx/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab.
WARNING: no policy specified for oryx/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab.
Entry for principal oryx/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab.
I20260620 23:02:06.402654 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30017
I20260620 23:02:06.407851 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30151
I20260620 23:02:06.412761 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30285
I20260620 23:02:06.417261 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 29923
I20260620 23:02:06.423338 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46345
--webserver_interface=127.21.66.126
--webserver_port=45489
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:06.506538 30423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:06.506786 30423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:06.506811 30423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:06.508056 30423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:06.508103 30423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:06.508118 30423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:06.508131 30423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:06.508147 30423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:06.509819 30423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=45489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:06.510154 30423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:06.510409 30423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:06.515483 30428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.515463 30429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.515465 30431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:06.515807 30423 server_base.cc:1034] running on GCE node
I20260620 23:02:06.516085 30423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:06.516405 30423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:06.517639 30423 hybrid_clock.cc:648] HybridClock initialized: now 1781996526517622 us; error 32 us; skew 500 ppm
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:06.520504 30423 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:02:06.521021 30423 webserver.cc:460] Webserver started at http://127.21.66.126:45489/ using document root <none> and password file <none>
I20260620 23:02:06.521266 30423 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:06.521353 30423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:06.522996 30423 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:02:06.523779 30438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.523963 30423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:06.524068 30423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "82a077732a464f1585808dfa06901cbb"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "a3933949b08bad68b414172817a7f1cf"
server_key_iv: "1439165567855ab62c65e3056074a06f"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.524494 30423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:06.537484 30423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:06.540305 30423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:06.540505 30423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:06.545594 30423 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:46345
I20260620 23:02:06.545632 30490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:46345 every 8 connection(s)
I20260620 23:02:06.546064 30423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:02:06.547608 30491 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:02:06.548297 30491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Bootstrap starting.
I20260620 23:02:06.548530 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30423
I20260620 23:02:06.548853 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:37299
--local_ip_for_outbound_sockets=127.21.66.65
--tserver_master_addrs=127.21.66.126:46345
--webserver_port=44455
--webserver_interface=127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:02:06.557140 30491 log.cc:824] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:06.559221 30491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:02:06.559551 30491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Bootstrap complete.
I20260620 23:02:06.561882 30491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:06.562170 30491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a077732a464f1585808dfa06901cbb, State: Initialized, Role: FOLLOWER
I20260620 23:02:06.562361 30491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:06.562441 30491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:06.562479 30491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:06.562526 30491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:02:06.563570 30491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:06.563694 30491 leader_election.cc:302] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82a077732a464f1585808dfa06901cbb; no voters:
I20260620 23:02:06.563905 30491 leader_election.cc:288] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:02:06.563943 30495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:02:06.564057 30495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [term 2 LEADER]: Becoming Leader. State: Replica: 82a077732a464f1585808dfa06901cbb, State: Running, Role: LEADER
I20260620 23:02:06.564147 30495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } }
I20260620 23:02:06.564203 30491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:06.564471 30497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "82a077732a464f1585808dfa06901cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } } }
I20260620 23:02:06.564600 30497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:06.564464 30496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82a077732a464f1585808dfa06901cbb. Latest consensus state: current_term: 2 leader_uuid: "82a077732a464f1585808dfa06901cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82a077732a464f1585808dfa06901cbb" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 46345 } } }
I20260620 23:02:06.564847 30496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:06.565052 30501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:06.565632 30501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:06.565794 30501 catalog_manager.cc:1259] Loaded cluster ID: b518eafe8c5949bf9ea9b8fa37beec27
I20260620 23:02:06.566030 30501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:06.566891 30501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:06.567091 30501 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 82a077732a464f1585808dfa06901cbb: Loaded TSK: 0
I20260620 23:02:06.567318 30501 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:02:06.648499 30493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:06.648732 30493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:06.648761 30493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:06.650149 30493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:06.650204 30493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:06.650235 30493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:02:06.651791 30493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.65.keytab
--principal=oryx/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:37299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=44455
--tserver_master_addrs=127.21.66.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:06.652062 30493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:06.652343 30493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:06.658937 30520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.658942 30518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.658954 30517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:06.659514 30493 server_base.cc:1034] running on GCE node
I20260620 23:02:06.659773 30493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:06.660037 30493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:06.661235 30493 hybrid_clock.cc:648] HybridClock initialized: now 1781996526661207 us; error 45 us; skew 500 ppm
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:06.664577 30493 init.cc:377] Logged in from keytab as oryx/127.21.66.65@KRBTEST.COM (short username oryx)
I20260620 23:02:06.665269 30493 webserver.cc:460] Webserver started at http://127.21.66.65:44455/ using document root <none> and password file <none>
I20260620 23:02:06.665553 30493 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:06.665628 30493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:06.667318 30493 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:06.668077 30527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.668305 30493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:06.668431 30493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "c2d24bae6b3c4cb498b402d58fcb0497"
format_stamp: "Formatted at 2026-06-20 23:02:05 on dist-test-slave-qgtk"
server_key: "fe31d6a2b35f485ae5fbb5fb0769fae9"
server_key_iv: "78293c9c7bb044c2c67ff180d7f414e0"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.668970 30493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:06.691248 30493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:06.694562 30493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:06.694749 30493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:06.695042 30493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:06.695456 30493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:06.695510 30493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.695555 30493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:06.695586 30493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.703970 30493 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:37299
I20260620 23:02:06.704035 30640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:37299 every 8 connection(s)
I20260620 23:02:06.704500 30493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:02:06.704941 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30493
I20260620 23:02:06.705343 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:37125
--local_ip_for_outbound_sockets=127.21.66.66
--tserver_master_addrs=127.21.66.126:46345
--webserver_port=42885
--webserver_interface=127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:06.714377 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.705399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:35431)
0620 23:02:06.705668 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.705754 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.706549 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.707277 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.707282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.707936 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.708139 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.708144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.709311 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.709312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.710230 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.710233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.710926 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.710928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.711063 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.711659 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.711668 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.714105 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
W20260620 23:02:06.714833 30641 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:46345 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.66.126:46345: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
W20260620 23:02:06.715153 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:06.705494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:35431 (local address 127.21.66.126:46345)
0620 23:02:06.706811 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.706821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.706866 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:02:06.707002 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.707010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.707070 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.707236 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.708302 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.709198 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.710321 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.710777 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.710954 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.714306 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:35431: BlockingRecv error: recv got EOF from 127.21.66.65:35431 (error 108)
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":95,"threads_started":1}
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:06.720021 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.715110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:49189)
0620 23:02:06.715365 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.715375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.715415 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.715690 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.715692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.715841 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.715895 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.715897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.716558 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.716559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.717066 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.717067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.717440 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.717441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.717511 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.717983 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.717988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.719928 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238}
W20260620 23:02:06.720079 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:06.715147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:49189 (local address 127.21.66.126:46345)
0620 23:02:06.715552 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.715554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.715562 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:06.715599 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.715600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.715610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.715673 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.715985 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.716464 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.717154 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.717351 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.717441 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.720011 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:49189: BlockingRecv error: recv got EOF from 127.21.66.65:49189 (error 108)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":118,"threads_started":1}
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:06.725195 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.720529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:53865)
0620 23:02:06.720608 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.720620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.720668 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.720825 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.720826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.720958 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.720989 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.720991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.721670 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.721671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.722267 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.722270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.722648 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.722649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.722714 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.723165 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.723171 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.725096 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 23:02:06.725257 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:06.720572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:53865 (local address 127.21.66.126:46345)
0620 23:02:06.720612 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.720614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.720652 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:02:06.720736 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.720737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.720748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.720820 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.721074 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.721590 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.722360 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.722546 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.722635 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.725187 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:53865: BlockingRecv error: recv got EOF from 127.21.66.65:53865 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:02:06.725651 30641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:02:06.813462 30645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:06.813704 30645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:06.813730 30645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:06.815112 30645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:06.815168 30645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:06.815197 30645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:02:06.816699 30645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.66.keytab
--principal=oryx/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:37125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=42885
--tserver_master_addrs=127.21.66.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:06.816918 30645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:06.817126 30645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:06.822431 30652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:06.822638 30645 server_base.cc:1034] running on GCE node
W20260620 23:02:06.822432 30654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.822444 30651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:06.822944 30645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:06.823216 30645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:06.824396 30645 hybrid_clock.cc:648] HybridClock initialized: now 1781996526824352 us; error 62 us; skew 500 ppm
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:06.827282 30645 init.cc:377] Logged in from keytab as oryx/127.21.66.66@KRBTEST.COM (short username oryx)
I20260620 23:02:06.827777 30645 webserver.cc:460] Webserver started at http://127.21.66.66:42885/ using document root <none> and password file <none>
I20260620 23:02:06.828121 30645 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:06.828207 30645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:06.829679 30645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:02:06.830421 30661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.830627 30645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:06.830729 30645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "748db8c8451547a9a05d2216b96f7995"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "270711a220de02cb22fc9ef6e8e3170c"
server_key_iv: "900f7211c45418230ddfd31065a29573"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.831099 30645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:06.840507 30645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:06.843123 30645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:06.843307 30645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:06.843564 30645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:06.843907 30645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:06.843963 30645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.844007 30645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:06.844040 30645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.850086 30645 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:37125
I20260620 23:02:06.850265 30774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:37125 every 8 connection(s)
I20260620 23:02:06.850570 30645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:02:06.852249 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30645
I20260620 23:02:06.852681 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:45701
--local_ip_for_outbound_sockets=127.21.66.67
--tserver_master_addrs=127.21.66.126:46345
--webserver_port=46619
--webserver_interface=127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:06.858722 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.851390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:60937)
0620 23:02:06.851558 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.851610 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.852092 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.852356 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.852361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.853080 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.853293 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.853299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.854218 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.854219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.855043 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.855046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.855555 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.855556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.855702 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.856293 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.856303 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.858555 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260620 23:02:06.858827 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:06.851398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:60937 (local address 127.21.66.126:46345)
0620 23:02:06.851478 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.851480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.851730 (+ 250us) server_negotiation.cc:407] Connection header received
0620 23:02:06.852208 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.852210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.852227 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.852278 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.853398 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.854057 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.855262 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.855448 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.855542 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.858726 (+ 3184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:60937: BlockingRecv error: recv got EOF from 127.21.66.66:60937 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 23:02:06.858994 30775 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:46345 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.66.126:46345: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:06.864423 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.859269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:37259)
0620 23:02:06.859362 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.859373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.859421 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.859611 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.859612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.859752 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.859797 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.859798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.860441 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.860442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.860954 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.860955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.861301 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.861302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.861366 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.861941 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.861946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.864329 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 23:02:06.864519 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:06.859312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:37259 (local address 127.21.66.126:46345)
0620 23:02:06.859358 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.859360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.859404 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:02:06.859521 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.859522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.859532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.859610 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.859875 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.860347 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.861037 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.861229 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.861304 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.864413 (+ 3109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:37259: BlockingRecv error: recv got EOF from 127.21.66.66:37259 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:06.869555 30777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.864727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:36157)
0620 23:02:06.864786 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.864797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.864845 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.864986 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.864987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.865112 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:06.865154 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.865155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.865908 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.865909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.866501 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:06.866503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.866907 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.866908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:06.866967 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.867497 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:06.867502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:06.869500 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 23:02:06.869706 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:06.864763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:36157 (local address 127.21.66.126:46345)
0620 23:02:06.864794 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.864797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.864871 (+ 74us) server_negotiation.cc:407] Connection header received
0620 23:02:06.864899 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.864900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.864909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.864980 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:06.865233 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.865817 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.866629 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:06.866819 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:06.866899 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:06.869629 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:36157: BlockingRecv error: recv got EOF from 127.21.66.66:36157 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 23:02:06.869786 30775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:02:06.947356 30778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:06.947592 30778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:06.947618 30778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:06.948904 30778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:06.948963 30778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:06.948992 30778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:02:06.950555 30778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/krb5kdc/oryx_127.21.66.67.keytab
--principal=oryx/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:45701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=46619
--tserver_master_addrs=127.21.66.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:06.950771 30778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:06.950975 30778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:06.956239 30784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.956327 30786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:06.956382 30783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:06.956715 30778 server_base.cc:1034] running on GCE node
I20260620 23:02:06.956907 30778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:06.957185 30778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:06.958389 30778 hybrid_clock.cc:648] HybridClock initialized: now 1781996526958363 us; error 42 us; skew 500 ppm
Jun 20 23:02:06 dist-test-slave-qgtk krb5kdc[29910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996526, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:06.961346 30778 init.cc:377] Logged in from keytab as oryx/127.21.66.67@KRBTEST.COM (short username oryx)
I20260620 23:02:06.961920 30778 webserver.cc:460] Webserver started at http://127.21.66.67:46619/ using document root <none> and password file <none>
I20260620 23:02:06.962247 30778 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:06.962337 30778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:06.963759 30778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:06.964540 30793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.964772 30778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:06.964876 30778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "4a9ed630a0e046b5bb3b7aaa73926d29"
format_stamp: "Formatted at 2026-06-20 23:02:06 on dist-test-slave-qgtk"
server_key: "13b12f04d09f6a84a1ef8a6a8859effd"
server_key_iv: "31221192903c47644c8148f7e67d0697"
server_key_version: "encryptionkey@0"
I20260620 23:02:06.965394 30778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:06.984926 30778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:06.987506 30778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:06.987704 30778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:06.988163 30778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:06.988731 30778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:06.988798 30778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.988847 30778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:06.988880 30778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:06.996043 30778 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:45701
I20260620 23:02:06.996140 30906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:45701 every 8 connection(s)
I20260620 23:02:06.996661 30778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:02:06.998879 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30778
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:02:07.006361 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:06.997869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:38709 (local address 127.21.66.126:46345)
0620 23:02:06.997966 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.997968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.998511 (+ 543us) server_negotiation.cc:407] Connection header received
0620 23:02:06.999136 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.999138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.999151 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.999211 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:07.000344 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.000953 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.002043 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.002330 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.002417 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.006294 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:38709: BlockingRecv error: recv got EOF from 127.21.66.67:38709 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260620 23:02:07.006376 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.998015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:38709)
0620 23:02:06.998346 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:06.998389 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:06.998963 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:06.999297 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:06.999302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:06.999984 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:07.000196 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.000201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.001089 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.001090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.001943 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.001946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.002429 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.002431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.002567 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.003065 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:07.003074 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:07.006157 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
W20260620 23:02:07.006938 30907 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:46345 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.66.126:46345: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
I20260620 23:02:07.007982 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:06.999503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.21.66.126:46345)
0620 23:02:06.999586 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:06.999589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:06.999599 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:02:06.999665 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:06.999666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:06.999678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:06.999778 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:07.000424 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.000988 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.001842 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.002063 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.002176 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.003765 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:07.003773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:07.003777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:07.003789 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:07.006034 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:07.006585 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:07.006590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:07.006591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:07.006734 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:07.007035 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:07.007036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:07.007037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:07.007418 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:07.007580 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:07.007728 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:02:07.007864 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:07.012823 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:07.007300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:35619)
0620 23:02:07.007442 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:07.007454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:07.007517 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:07.007657 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:07.007658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:07.007818 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:07.007876 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.007877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.008741 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.008742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.009388 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.009390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.009746 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.009747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.009811 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.010417 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:07.010422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:07.012733 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20260620 23:02:07.012897 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:07.007386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:35619 (local address 127.21.66.126:46345)
0620 23:02:07.007443 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:07.007445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:07.007501 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:02:07.007572 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:07.007573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:07.007583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:07.007654 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:07.007994 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.008668 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.009476 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.009675 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.009750 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.012847 (+ 3097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:35619: BlockingRecv error: recv got EOF from 127.21.66.67:35619 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:07.017746 30909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:07.013332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:35755)
0620 23:02:07.013403 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:07.013415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:07.013471 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:07.013581 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:07.013581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:07.013691 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:07.013723 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.013724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.014499 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.014500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.015044 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.015045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.015424 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.015425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.015474 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.015933 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:07.015937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:07.017684 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:02:07.017897 30644 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:07.013411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:35755 (local address 127.21.66.126:46345)
0620 23:02:07.013469 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:07.013470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:07.013478 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:07.013518 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:07.013518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:07.013527 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:07.013566 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:07.013810 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.014415 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.015149 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.015351 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.015413 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.017805 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:35755: BlockingRecv error: recv got EOF from 127.21.66.67:35755 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 23:02:07.017946 30907 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:07.731233 30911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:07.726168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:36993)
0620 23:02:07.726311 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:07.726325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:07.726379 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:07.726527 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:07.726527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:07.726706 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:07.726754 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.726755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.727473 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.727473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.728028 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.728030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.728519 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.728519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.728563 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.729047 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:07.729052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:07.731123 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260620 23:02:07.731272 30912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:07.726238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:36993 (local address 127.21.66.126:46345)
0620 23:02:07.726393 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:07.726395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:07.726405 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:02:07.726432 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:07.726432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:07.726442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:07.726509 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:07.726811 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.727324 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.728150 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.728437 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.728511 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.731200 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:36993: BlockingRecv error: recv got EOF from 127.21.66.65:36993 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:07 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:07.876550 30913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:07.870223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:43409)
0620 23:02:07.870485 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:07.870500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:07.870609 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:07.870861 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:07.870862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:07.871078 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:07.871130 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.871133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.871960 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.871962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.872675 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:07.872678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.873150 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.873152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:07.873259 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.873915 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:07.873922 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:07.876438 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":58,"threads_started":1}
W20260620 23:02:07.876592 30912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:07.870532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:43409 (local address 127.21.66.126:46345)
0620 23:02:07.870604 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:07.870607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:07.870620 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:02:07.870751 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:07.870752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:07.870767 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:07.870854 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:07.871251 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.871857 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.872802 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:07.873027 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:07.873136 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:07.876528 (+ 3392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:43409: BlockingRecv error: recv got EOF from 127.21.66.66:43409 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 23:02:08 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:08 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:08.023708 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:08.018552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:34083)
0620 23:02:08.018725 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:08.018739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:08.018822 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:08.018975 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:08.018976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:08.019153 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:08.019227 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:08.019228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.020188 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.020189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:08.020764 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:08.020765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.021159 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.021160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:08.021224 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:08.021744 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:08.021750 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:08.023597 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
W20260620 23:02:08.023746 30912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:08.018692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:34083 (local address 127.21.66.126:46345)
0620 23:02:08.018772 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:08.018774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:08.018805 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:02:08.018888 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:08.018889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:08.018900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:08.018952 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:08.019341 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.020121 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.020865 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.021072 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.021158 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:08.023679 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:34083: BlockingRecv error: recv got EOF from 127.21.66.67:34083 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 23:02:08 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:08 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:08.736853 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:08.731796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:36919)
0620 23:02:08.731928 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:08.731946 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:08.731998 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:08.732276 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:08.732276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:08.732443 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:08.732494 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:08.732495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.733143 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.733144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:08.733759 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:08.733764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.734309 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.734310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:08.734378 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:08.734946 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:08.734951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:08.736792 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
W20260620 23:02:08.737000 30916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:08.732005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:36919 (local address 127.21.66.126:46345)
0620 23:02:08.732142 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:08.732144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:08.732153 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:08.732178 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:08.732179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:08.732190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:08.732259 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:08.732574 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.733063 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.733871 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.734214 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.734322 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:08.736901 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:36919: BlockingRecv error: recv got EOF from 127.21.66.65:36919 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
Jun 20 23:02:08 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:08 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:08.883206 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:08.877150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:47931)
0620 23:02:08.877387 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:08.877404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:08.877479 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:08.877687 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:08.877688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:08.877902 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:08.877955 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:08.877958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.879076 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.879078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:08.879704 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:08.879708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.880057 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.880058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:08.880336 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:08.881045 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:08.881050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:08.883109 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
W20260620 23:02:08.883283 30916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:08.877261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:47931 (local address 127.21.66.126:46345)
0620 23:02:08.877448 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:08.877450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:08.877484 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:02:08.877568 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:08.877569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:08.877585 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:08.877680 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:08.878058 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.878954 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.879796 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:08.879983 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:08.880049 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:08.883208 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:47931: BlockingRecv error: recv got EOF from 127.21.66.66:47931 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 23:02:09 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:09 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:09.029781 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:09.024278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:42671)
0620 23:02:09.024439 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:09.024449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:09.024523 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:09.024656 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:09.024657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:09.024835 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:09.024882 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:09.024884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.025668 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.025669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:09.026275 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:09.026277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.026684 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.026685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:09.026745 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:09.027230 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:09.027235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:09.029720 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260620 23:02:09.029969 30916 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:09.024467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:42671 (local address 127.21.66.126:46345)
0620 23:02:09.024535 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:09.024537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:09.024553 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:02:09.024581 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:09.024582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:09.024596 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:09.024651 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:09.024977 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.025562 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.026377 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.026608 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.026682 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:09.029857 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:42671: BlockingRecv error: recv got EOF from 127.21.66.67:42671 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 23:02:09 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:09 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:09.742506 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:09.737528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:57309)
0620 23:02:09.737705 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:09.737720 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:09.737802 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:09.737950 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:09.737951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:09.738167 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:09.738222 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:09.738224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.738940 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.738941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:09.739531 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:09.739533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.739942 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.739942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:09.739985 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:09.740447 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:09.740451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:09.742442 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
W20260620 23:02:09.742676 30920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:09.737648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:57309 (local address 127.21.66.126:46345)
0620 23:02:09.737807 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:09.737809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:09.737837 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:02:09.737862 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:09.737863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:09.737874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:09.737925 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:09.738296 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.738852 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.739669 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.739867 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.739927 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:09.742567 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:57309: BlockingRecv error: recv got EOF from 127.21.66.65:57309 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
Jun 20 23:02:09 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:09 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:09.889055 30921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:09.883807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:57869)
0620 23:02:09.883968 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:09.883988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:09.884063 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:09.884225 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:09.884227 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:09.884397 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:09.884447 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:09.884449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.885165 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.885166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:09.885787 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:09.885789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.886163 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.886164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:09.886288 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:09.886784 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:09.886789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:09.888988 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
W20260620 23:02:09.889271 30920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:09.883993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:57869 (local address 127.21.66.126:46345)
0620 23:02:09.884068 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:09.884070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:09.884083 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:02:09.884114 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:09.884115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:09.884126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:09.884185 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:09.884525 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.885079 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.885877 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:09.886067 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:09.886208 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:09.889166 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:57869: BlockingRecv error: recv got EOF from 127.21.66.66:57869 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 23:02:10 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:10 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:10.035863 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:10.030463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:34141)
0620 23:02:10.030738 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:10.030755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:10.030868 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:10.031069 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:10.031071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:10.031279 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:10.031328 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:10.031330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.032140 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.032141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:10.032730 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:10.032732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.033132 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.033133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:10.033207 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:10.033683 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:10.033687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:10.035777 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":162,"threads_started":1}
W20260620 23:02:10.035874 30920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:10.030663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:34141 (local address 127.21.66.126:46345)
0620 23:02:10.030757 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:10.030760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:10.030846 (+ 86us) server_negotiation.cc:407] Connection header received
0620 23:02:10.030966 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:10.030968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:10.030990 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:10.031061 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:10.031427 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.032044 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.032824 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.033036 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.033130 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:10.035815 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:34141: BlockingRecv error: recv got EOF from 127.21.66.67:34141 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 23:02:10 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:10 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:10.752544 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:10.743167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:42909)
0620 23:02:10.743373 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:10.743386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:10.743452 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:10.743634 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:10.743634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:10.743870 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:10.743922 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:10.743923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.744593 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.744594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:10.745207 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:10.745209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.745599 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.745600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:10.745670 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:10.750395 (+ 4725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:10.750401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:10.752478 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":116,"threads_started":1}
W20260620 23:02:10.752828 30924 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:10.743362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:42909 (local address 127.21.66.126:46345)
0620 23:02:10.743505 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:10.743507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:10.743516 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:10.743541 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:10.743542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:10.743551 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:10.743613 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:10.744011 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.744520 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.745292 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.745483 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.745579 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:10.752766 (+ 7187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:42909: BlockingRecv error: recv got EOF from 127.21.66.65:42909 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
Jun 20 23:02:10 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:10 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:10.894956 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:10.889663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:51473)
0620 23:02:10.889884 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:10.889900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:10.890029 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:10.890243 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:10.890244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:10.890442 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:10.890490 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:10.890492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.891259 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.891260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:10.891892 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:10.891895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.892277 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.892278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:10.892353 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:10.892864 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:10.892870 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:10.894895 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
W20260620 23:02:10.895140 30924 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:10.889778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:51473 (local address 127.21.66.126:46345)
0620 23:02:10.889901 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:10.889904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:10.889983 (+ 79us) server_negotiation.cc:407] Connection header received
0620 23:02:10.890105 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:10.890106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:10.890149 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:10.890214 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:10.890595 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.891157 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.891989 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:10.892199 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:10.892280 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:10.895041 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:51473: BlockingRecv error: recv got EOF from 127.21.66.66:51473 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 23:02:11 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:11 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:11.041021 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:11.036356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:33321)
0620 23:02:11.036497 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:11.036513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:11.036568 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:11.036747 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:11.036748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:11.036927 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:11.036977 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:11.036979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.037688 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.037689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:11.038240 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:11.038242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.038666 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.038667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:11.038706 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:11.039218 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:11.039223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:11.040961 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
W20260620 23:02:11.041191 30924 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:11.036441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:33321 (local address 127.21.66.126:46345)
0620 23:02:11.036512 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:11.036515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:11.036559 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:02:11.036627 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:11.036628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:11.036641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:11.036692 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:11.037064 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.037603 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.038352 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.038587 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.038660 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:11.041101 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:33321: BlockingRecv error: recv got EOF from 127.21.66.67:33321 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 23:02:11 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:11 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:11.758926 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:11.753109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:39897)
0620 23:02:11.753404 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:11.753420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:11.753489 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:11.753632 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:11.753633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:11.753817 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:11.753864 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:11.753866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.754738 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.754739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:11.755274 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:11.755276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.755717 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.755718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:11.755774 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:11.756381 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:11.756386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:11.758828 (+ 2442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":202,"threads_started":1}
W20260620 23:02:11.758996 30928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:11.753312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:39897 (local address 127.21.66.126:46345)
0620 23:02:11.753436 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:11.753438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:11.753473 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:02:11.753549 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:11.753549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:11.753559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:11.753611 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:11.754011 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.754632 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.755368 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.755608 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.755682 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:11.758924 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:39897: BlockingRecv error: recv got EOF from 127.21.66.65:39897 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
Jun 20 23:02:11 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:11 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:11.900303 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:11.895490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:37833)
0620 23:02:11.895629 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:11.895646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:11.895695 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:11.895861 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:11.895862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:11.896024 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:11.896071 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:11.896073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.896882 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.896883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:11.897436 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:11.897439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.897839 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.897840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:11.897881 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:11.898472 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:11.898476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:11.900233 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260620 23:02:11.900520 30928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:11.895634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:37833 (local address 127.21.66.126:46345)
0620 23:02:11.895690 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:11.895692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:11.895700 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:11.895773 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:11.895774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:11.895785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:11.895839 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:11.896205 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.896753 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.897526 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:11.897752 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:11.897821 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:11.900451 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:37833: BlockingRecv error: recv got EOF from 127.21.66.66:37833 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 23:02:12 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:12 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:12.046954 30930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:12.041582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:56903)
0620 23:02:12.041790 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:12.041803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:12.041885 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:12.042109 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:12.042139 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:12.042308 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:12.042417 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:12.042420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.043114 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.043115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:12.043742 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:12.043744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.044076 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.044077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:12.044119 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:12.044689 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:12.044695 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:12.046889 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
W20260620 23:02:12.047524 30928 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:12.041634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:56903 (local address 127.21.66.126:46345)
0620 23:02:12.041720 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:12.041722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:12.041898 (+ 176us) server_negotiation.cc:407] Connection header received
0620 23:02:12.041979 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:12.041980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:12.041998 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:12.042085 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:12.042532 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.043031 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.043828 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.044006 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.044058 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:12.047304 (+ 3246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:56903: BlockingRecv error: recv got EOF from 127.21.66.67:56903 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 23:02:12 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:12 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:12.764545 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:12.759551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:46011)
0620 23:02:12.759719 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:12.759737 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:12.759797 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:12.759953 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:12.759954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:12.760113 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:12.760162 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:12.760163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.760834 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.760835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:12.761374 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:12.761376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.761716 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.761717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:12.761764 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:12.762296 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:12.762305 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:12.764443 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
W20260620 23:02:12.764608 30932 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:12.759644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:46011 (local address 127.21.66.126:46345)
0620 23:02:12.759825 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:12.759827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:12.759835 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:12.759860 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:12.759861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:12.759871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:12.759931 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:12.760243 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.760753 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.761457 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.761649 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.761705 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:12.764536 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:46011: BlockingRecv error: recv got EOF from 127.21.66.65:46011 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":52,"threads_started":1}
Jun 20 23:02:12 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:12 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:12.905934 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:12.900916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:48657)
0620 23:02:12.901071 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:12.901085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:12.901168 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:12.901323 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:12.901324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:12.901522 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:12.901584 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:12.901586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.902251 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.902252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:12.902880 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:12.902882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.903278 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.903279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:12.903330 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:12.903800 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:12.903805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:12.905865 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
W20260620 23:02:12.906020 30932 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:12.901001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:48657 (local address 127.21.66.126:46345)
0620 23:02:12.901090 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:12.901092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:12.901164 (+ 72us) server_negotiation.cc:407] Connection header received
0620 23:02:12.901236 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:12.901237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:12.901247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:12.901318 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:12.901662 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.902183 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.902970 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:12.903182 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:12.903238 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:12.905971 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:48657: BlockingRecv error: recv got EOF from 127.21.66.66:48657 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:02:13 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:13 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:13.052968 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:13.047836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:42923)
0620 23:02:13.047997 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:13.048007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:13.048067 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:13.048226 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:13.048227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:13.048387 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:13.048435 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:13.048437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.049145 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.049147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:13.049777 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:13.049780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.050262 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.050263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:13.050320 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:13.050842 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:13.050847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:13.052902 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
W20260620 23:02:13.053046 30932 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:13.047972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:42923 (local address 127.21.66.126:46345)
0620 23:02:13.048027 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:13.048028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:13.048055 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:02:13.048129 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:13.048130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:13.048147 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:13.048196 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:13.048540 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.049048 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.049895 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.050076 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.050159 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:13.052998 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:42923: BlockingRecv error: recv got EOF from 127.21.66.67:42923 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 23:02:13 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:13 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:13.770850 30935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:13.765190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:39171)
0620 23:02:13.765322 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:13.765356 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:13.765424 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:13.765755 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:13.765756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:13.766011 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:13.766074 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:13.766076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.767120 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.767122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:13.767739 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:13.767741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.768283 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.768285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:13.768373 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:13.768865 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:13.768870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:13.770738 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260620 23:02:13.771085 30936 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:13.765412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:39171 (local address 127.21.66.126:46345)
0620 23:02:13.765568 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:13.765570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:13.765581 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:02:13.765617 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:13.765618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:13.765632 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:13.765717 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:13.766203 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.767022 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.767847 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.768164 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.768279 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:13.770877 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:39171: BlockingRecv error: recv got EOF from 127.21.66.65:39171 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
Jun 20 23:02:13 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:13 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:13.911525 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:13.906529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:34379)
0620 23:02:13.906684 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:13.906699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:13.906751 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:13.906898 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:13.906898 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:13.907059 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:13.907105 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:13.907107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.907862 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.907863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:13.908395 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:13.908397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.908800 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.908801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:13.908841 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:13.909357 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:13.909362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:13.911426 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
W20260620 23:02:13.911616 30936 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:13.906675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:34379 (local address 127.21.66.126:46345)
0620 23:02:13.906747 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:13.906749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:13.906758 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:13.906807 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:13.906808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:13.906818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:13.906876 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:13.907202 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.907771 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.908488 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:13.908727 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:13.908791 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:13.911514 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:34379: BlockingRecv error: recv got EOF from 127.21.66.66:34379 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 23:02:14 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:14 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:14.058862 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:14.053663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:53555)
0620 23:02:14.053821 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:14.053834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:14.053897 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:14.054157 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:14.054159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:14.054383 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:14.054434 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:14.054435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.055107 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.055107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:14.055711 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:14.055713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.056055 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.056056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:14.056096 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:14.056640 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:14.056645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:14.058760 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
W20260620 23:02:14.058914 30936 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:14.053788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:53555 (local address 127.21.66.126:46345)
0620 23:02:14.053868 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:14.053870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:14.053881 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:02:14.054008 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:14.054009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:14.054023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:14.054102 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:14.054520 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.055024 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.055798 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.055987 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.056044 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:14.058848 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:53555: BlockingRecv error: recv got EOF from 127.21.66.67:53555 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 23:02:14 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:14 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:14.777299 30939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:14.771393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:46347)
0620 23:02:14.771675 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:14.771692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:14.771785 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:14.772051 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:14.772052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:14.772224 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:14.772275 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:14.772277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.773025 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.773026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:14.773913 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:14.773916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.774354 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.774355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:14.774425 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:14.774986 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:14.774992 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:14.777228 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":78,"threads_started":1}
W20260620 23:02:14.777449 30940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:14.771674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:46347 (local address 127.21.66.126:46345)
0620 23:02:14.771838 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:14.771840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:14.771859 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:02:14.771888 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:14.771889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:14.771900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:14.772013 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:14.772373 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.772950 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.773977 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.774275 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.774342 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:14.777354 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:46347: BlockingRecv error: recv got EOF from 127.21.66.65:46347 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
Jun 20 23:02:14 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:14 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:14.917382 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:14.912236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:34069)
0620 23:02:14.912387 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:14.912406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:14.912473 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:14.912660 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:14.912661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:14.912826 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:14.912875 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:14.912877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.913709 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.913710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:14.914301 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:14.914302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.914751 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.914752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:14.914803 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:14.915384 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:14.915388 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:14.917320 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
W20260620 23:02:14.917519 30940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:14.912305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:34069 (local address 127.21.66.126:46345)
0620 23:02:14.912400 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:14.912402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:14.912479 (+ 77us) server_negotiation.cc:407] Connection header received
0620 23:02:14.912531 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:14.912532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:14.912542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:14.912605 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:14.912951 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.913600 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.914387 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:14.914578 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:14.914645 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:14.917434 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:34069: BlockingRecv error: recv got EOF from 127.21.66.66:34069 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 23:02:15 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:15 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:15.065598 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:15.059451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:35075)
0620 23:02:15.059624 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:15.059638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:15.059723 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:15.059880 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:15.059881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:15.060122 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:15.060187 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:15.060189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.061059 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.061061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:15.062011 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:15.062015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.062449 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.062450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:15.062518 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:15.062995 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:15.063001 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:15.065478 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
W20260620 23:02:15.065845 30940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:15.059516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:35075 (local address 127.21.66.126:46345)
0620 23:02:15.059580 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:15.059583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:15.059721 (+ 138us) server_negotiation.cc:407] Connection header received
0620 23:02:15.059784 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:15.059785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:15.059798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:15.059852 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:15.060274 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.060982 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.062157 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.062370 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.062428 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:15.065773 (+ 3345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:35075: BlockingRecv error: recv got EOF from 127.21.66.67:35075 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 23:02:15 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:15 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:15.787303 30943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:15.777774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:51151)
0620 23:02:15.777921 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:15.777936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:15.778107 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:15.778308 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:15.778309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:15.778481 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:15.778534 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:15.778537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.779210 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.779211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:15.779767 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:15.779769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.780262 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.780263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:15.780310 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:15.785153 (+ 4843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:15.785170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:15.787242 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260620 23:02:15.787463 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:15.777883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:51151 (local address 127.21.66.126:46345)
0620 23:02:15.778037 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:15.778039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:15.778085 (+ 46us) server_negotiation.cc:407] Connection header received
0620 23:02:15.778209 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:15.778211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:15.778221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:15.778279 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:15.778598 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.779121 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.779869 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.780177 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.780241 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:15.787370 (+ 7129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:51151: BlockingRecv error: recv got EOF from 127.21.66.65:51151 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
Jun 20 23:02:15 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:15 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:15.923529 30945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:15.917949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:34371)
0620 23:02:15.918208 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:15.918225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:15.918290 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:15.918533 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:15.918534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:15.918809 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:15.918884 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:15.918886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.919783 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.919784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:15.920388 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:15.920390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.920714 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.920714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:15.920756 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:15.921296 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:15.921302 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:15.923451 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
W20260620 23:02:15.923720 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:15.918054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:34371 (local address 127.21.66.126:46345)
0620 23:02:15.918172 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:15.918174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:15.918292 (+ 118us) server_negotiation.cc:407] Connection header received
0620 23:02:15.918374 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:15.918375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:15.918390 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:15.918475 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:15.919015 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.919691 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.920469 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:15.920648 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:15.920700 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:15.923634 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:34371: BlockingRecv error: recv got EOF from 127.21.66.66:34371 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 23:02:16 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:16 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:16.071724 30946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:16.066253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:52549)
0620 23:02:16.066397 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:16.066408 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:16.066470 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:16.066601 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:16.066602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:16.066775 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:16.066825 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:16.066827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.067606 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.067608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:16.068284 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:16.068287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.068683 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.068684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:16.068737 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:16.069379 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:16.069385 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:16.071624 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260620 23:02:16.071782 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:16.066352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:52549 (local address 127.21.66.126:46345)
0620 23:02:16.066427 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:16.066429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:16.066458 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:02:16.066516 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:16.066517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:16.066530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:16.066591 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:16.066924 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.067499 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.068408 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.068609 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.068668 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:16.071713 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:52549: BlockingRecv error: recv got EOF from 127.21.66.67:52549 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 23:02:16 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:16 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:16.792917 30947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:16.787813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:51639)
0620 23:02:16.787957 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:16.787967 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:16.788017 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:16.788216 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:16.788217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:16.788383 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:16.788433 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:16.788435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.789140 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.789141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:16.789703 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:16.789705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.790196 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.790197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:16.790245 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:16.790858 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:16.790862 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:16.792805 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
W20260620 23:02:16.792971 30948 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:16.787950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:51639 (local address 127.21.66.126:46345)
0620 23:02:16.788079 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:16.788081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:16.788089 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:16.788112 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:16.788113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:16.788122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:16.788211 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:16.788505 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.789026 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.789815 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.790020 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.790093 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:16.792896 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:51639: BlockingRecv error: recv got EOF from 127.21.66.65:51639 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
Jun 20 23:02:16 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:16 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:16.929457 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:16.924128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:34211)
0620 23:02:16.924328 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:16.924348 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:16.924422 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:16.924561 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:16.924562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:16.924792 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:16.924843 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:16.924845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.925513 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.925514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:16.926097 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:16.926099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.926568 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.926569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:16.926691 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:16.927211 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:16.927217 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:16.929395 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
W20260620 23:02:16.929589 30948 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:16.924279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:34211 (local address 127.21.66.126:46345)
0620 23:02:16.924354 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:16.924356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:16.924404 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:02:16.924473 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:16.924474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:16.924485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:16.924536 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:16.924913 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.925427 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.926244 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:16.926462 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:16.926530 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:16.929506 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:34211: BlockingRecv error: recv got EOF from 127.21.66.66:34211 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 23:02:17 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:17 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:17.077765 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:17.072338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:34327)
0620 23:02:17.072531 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:17.072542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:17.072605 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:17.072793 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:17.072794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:17.072975 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:17.073026 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:17.073027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.074156 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.074158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:17.074797 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:17.074800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.075227 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.075228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:17.075281 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:17.075775 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:17.075780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:17.077692 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
W20260620 23:02:17.078018 30948 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:17.072544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:34327 (local address 127.21.66.126:46345)
0620 23:02:17.072621 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:17.072623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:17.072632 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:17.072659 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:17.072659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:17.072670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:17.072776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:17.073110 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.074024 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.074889 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.075082 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.075143 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:17.077957 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:34327: BlockingRecv error: recv got EOF from 127.21.66.67:34327 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 23:02:17 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:17 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:17.798534 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:17.793527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:48897)
0620 23:02:17.793664 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:17.793676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:17.793736 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:17.794037 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:17.794038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:17.794254 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:17.794302 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:17.794304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.795033 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.795034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:17.795593 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:17.795595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.795976 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.795977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:17.796033 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:17.796550 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:17.796554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:17.798467 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
W20260620 23:02:17.798722 30952 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:17.793660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:48897 (local address 127.21.66.126:46345)
0620 23:02:17.793858 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:17.793860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:17.793868 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:17.793891 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:17.793892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:17.793902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:17.793972 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:17.794403 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.794908 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.795684 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.795893 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.795985 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:17.798630 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:48897: BlockingRecv error: recv got EOF from 127.21.66.65:48897 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
Jun 20 23:02:17 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:17 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:17.936468 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:17.930074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:49291)
0620 23:02:17.930255 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:17.930277 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:17.930346 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:17.930561 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:17.930562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:17.930757 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:17.930806 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:17.930808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.931620 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.931621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:17.932265 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:17.932268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.933022 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.933023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:17.933074 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:17.933990 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:17.933996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:17.936395 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
W20260620 23:02:17.936561 30952 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:17.930307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:49291 (local address 127.21.66.126:46345)
0620 23:02:17.930390 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:17.930392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:17.930403 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:02:17.930437 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:17.930438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:17.930449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:17.930510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:17.930911 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.931505 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.932447 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:17.932922 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:17.932999 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:17.936509 (+ 3510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:49291: BlockingRecv error: recv got EOF from 127.21.66.66:49291 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 23:02:18 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:18 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:18.083775 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:18.078335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:39441)
0620 23:02:18.078492 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:18.078508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:18.078574 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:18.078761 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:18.078762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:18.078960 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:18.079032 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:18.079034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.079875 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.079876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:18.080568 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:18.080571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.080920 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.080921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:18.080970 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:18.081451 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:18.081456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:18.083691 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
W20260620 23:02:18.083853 30952 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:18.078428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:39441 (local address 127.21.66.126:46345)
0620 23:02:18.078494 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:18.078496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:18.078569 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:02:18.078623 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:18.078624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:18.078635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:18.078697 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:18.079120 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.079747 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.080655 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.080844 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.080893 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:18.083802 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:39441: BlockingRecv error: recv got EOF from 127.21.66.67:39441 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 23:02:18 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:18 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:18.803778 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:18.799060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:38413)
0620 23:02:18.799203 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:18.799216 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:18.799293 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:18.799447 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:18.799448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:18.799624 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:18.799673 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:18.799674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.800381 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.800382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:18.800917 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:18.800918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.801277 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.801278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:18.801321 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:18.801749 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:18.801753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:18.803720 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":52,"threads_started":1}
W20260620 23:02:18.803987 30956 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:18.799156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:38413 (local address 127.21.66.126:46345)
0620 23:02:18.799303 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:18.799305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:18.799312 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:02:18.799335 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:18.799335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:18.799345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:18.799405 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:18.799762 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.800310 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.800996 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.801209 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.801269 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:18.803886 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:38413: BlockingRecv error: recv got EOF from 127.21.66.65:38413 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
Jun 20 23:02:18 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:18 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:18.942318 30957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:18.937203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:37779)
0620 23:02:18.937343 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:18.937358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:18.937421 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:18.937589 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:18.937590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:18.937763 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:18.937809 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:18.937811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.938695 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.938696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:18.939224 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:18.939226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.939608 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.939609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:18.939649 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:18.940130 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:18.940134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:18.942256 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260620 23:02:18.942487 30956 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:18.937282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:37779 (local address 127.21.66.126:46345)
0620 23:02:18.937375 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:18.937377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:18.937414 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:02:18.937474 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:18.937475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:18.937488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:18.937567 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:18.937906 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.938604 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.939343 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:18.939535 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:18.939600 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:18.942394 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:37779: BlockingRecv error: recv got EOF from 127.21.66.66:37779 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 23:02:19 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:19 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:19.089830 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:19.084435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:50313)
0620 23:02:19.084611 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:19.084623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:19.084693 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:19.085032 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:19.085033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:19.085200 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:19.085247 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:19.085248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.086158 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.086159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:19.086878 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:19.086881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.087235 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.087235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:19.087296 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:19.087811 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:19.087816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:19.089765 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
W20260620 23:02:19.090054 30956 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:19.084629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:50313 (local address 127.21.66.126:46345)
0620 23:02:19.084772 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:19.084775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:19.084785 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:02:19.084927 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:19.084928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:19.084945 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:19.085000 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:19.085314 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.086006 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.086960 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.087164 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.087227 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:19.089969 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:50313: BlockingRecv error: recv got EOF from 127.21.66.67:50313 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 23:02:19 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:19 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:19.809813 30959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:19.804405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:50639)
0620 23:02:19.804641 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:19.804655 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:19.804735 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:19.804898 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:19.804899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:19.805239 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:19.805288 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:19.805292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.806102 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.806103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:19.806792 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:19.806815 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.807374 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.807375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:19.807426 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:19.807894 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:19.807898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:19.809742 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
W20260620 23:02:19.809998 30960 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:19.804523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:50639 (local address 127.21.66.126:46345)
0620 23:02:19.804665 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:19.804667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:19.804738 (+ 71us) server_negotiation.cc:407] Connection header received
0620 23:02:19.804789 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:19.804789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:19.804801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:19.804869 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:19.805376 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.805996 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.806899 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.807252 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.807351 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:19.809914 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:50639: BlockingRecv error: recv got EOF from 127.21.66.65:50639 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
Jun 20 23:02:19 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:19 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:19.948266 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:19.942914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:55017)
0620 23:02:19.943096 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:19.943112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:19.943182 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:19.943350 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:19.943351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:19.943531 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:19.943587 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:19.943589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.944358 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.944359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:19.944972 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:19.944975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.945424 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.945425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:19.945475 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:19.946018 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:19.946023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:19.948185 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
W20260620 23:02:19.948410 30960 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:19.943086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:55017 (local address 127.21.66.126:46345)
0620 23:02:19.943218 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:19.943220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:19.943229 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:19.943255 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:19.943255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:19.943266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:19.943322 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:19.943660 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.944252 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.945065 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:19.945334 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:19.945399 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:19.948311 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:55017: BlockingRecv error: recv got EOF from 127.21.66.66:55017 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Jun 20 23:02:20 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:20 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:20.095984 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:20.090453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:45697)
0620 23:02:20.090659 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:20.090670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:20.090743 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:20.090971 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:20.090972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:20.091156 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:20.091241 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:20.091243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.091919 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.091920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:20.092590 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:20.092593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.092980 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.092981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:20.093037 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:20.093598 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:20.093603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:20.095852 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
W20260620 23:02:20.096050 30960 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:20.090573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:45697 (local address 127.21.66.126:46345)
0620 23:02:20.090624 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:20.090626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:20.090792 (+ 166us) server_negotiation.cc:407] Connection header received
0620 23:02:20.090824 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:20.090825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:20.090838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:20.090890 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:20.091350 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.091835 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.092693 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.092884 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.092943 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:20.095981 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:45697: BlockingRecv error: recv got EOF from 127.21.66.67:45697 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 23:02:20 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:20 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:20.820735 30963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:20.810369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:38533)
0620 23:02:20.810521 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:20.810534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:20.810600 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:20.810730 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:20.810731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:20.810896 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:20.810943 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:20.810944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.811690 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.811692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:20.812244 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:20.812245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.812624 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.812626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:20.812687 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:20.818794 (+ 6107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:20.818800 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:20.820674 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
W20260620 23:02:20.820919 30964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:20.810424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:38533 (local address 127.21.66.126:46345)
0620 23:02:20.810587 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:20.810589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:20.810598 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:20.810647 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:20.810648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:20.810657 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:20.810716 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:20.811036 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.811579 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.812332 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.812552 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.812613 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:20.820831 (+ 8218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:38533: BlockingRecv error: recv got EOF from 127.21.66.65:38533 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
Jun 20 23:02:20 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:20 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:20.954267 30965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:20.948910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:44343)
0620 23:02:20.949176 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:20.949187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:20.949264 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:20.949423 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:20.949423 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:20.949622 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:20.949673 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:20.949675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.950398 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.950399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:20.951013 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:20.951015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.951422 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.951423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:20.951480 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:20.951950 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:20.951955 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:20.954165 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":147,"threads_started":1}
W20260620 23:02:20.954314 30964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:20.949091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:44343 (local address 127.21.66.126:46345)
0620 23:02:20.949180 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:20.949182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:20.949266 (+ 84us) server_negotiation.cc:407] Connection header received
0620 23:02:20.949323 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:20.949324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:20.949336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:20.949393 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:20.949769 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.950302 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.951095 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:20.951309 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:20.951375 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:20.954263 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:44343: BlockingRecv error: recv got EOF from 127.21.66.66:44343 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:02:21 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:21 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.67@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:21.101917 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:21.096531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.67:48609)
0620 23:02:21.096717 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:21.096831 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:21.096896 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:21.097130 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:21.097131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:21.097428 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:21.097490 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:21.097492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.098232 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.098233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:21.098771 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:21.098773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.099184 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.099185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:21.099310 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:21.099913 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:21.099919 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:21.101852 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
W20260620 23:02:21.102066 30964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:21.096705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:48609 (local address 127.21.66.126:46345)
0620 23:02:21.096770 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:21.096773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:21.096913 (+ 140us) server_negotiation.cc:407] Connection header received
0620 23:02:21.096952 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:21.096952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:21.096969 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:21.097066 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:21.097565 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.098066 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.098858 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.099101 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.099180 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:21.101983 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.67:48609: BlockingRecv error: recv got EOF from 127.21.66.67:48609 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 23:02:21 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:21 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.65@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:21.826681 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:21.821301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.65:60059)
0620 23:02:21.821473 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:21.821486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:21.821553 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:21.821755 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:21.821756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:21.821919 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:21.821976 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:21.821978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.822794 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.822795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:21.823380 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:21.823382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.823754 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.823755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:21.823801 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:21.824307 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:21.824311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:21.826537 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
W20260620 23:02:21.826720 30968 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:21.821369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:60059 (local address 127.21.66.126:46345)
0620 23:02:21.821529 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:21.821531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:21.821540 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:02:21.821622 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:21.821622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:21.821633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:21.821712 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:21.822054 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.822704 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.823483 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.823682 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.823739 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:21.826635 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.65:60059: BlockingRecv error: recv got EOF from 127.21.66.65:60059 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
Jun 20 23:02:21 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:02:21 dist-test-slave-qgtk krb5kdc[29910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.66.66@KRBTEST.COM for oryx/127.21.66.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:02:21.959859 30969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:21.954878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:46345 (local address 127.21.66.66:33207)
0620 23:02:21.955119 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:21.955131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:21.955197 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:21.955398 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:21.955398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:21.955568 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:21.955616 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:21.955618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.956413 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.956413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:21.956954 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:21.956956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.957354 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.957355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:21.957402 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:21.957917 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:21.957921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:21.959798 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.66.126:46345: Server oryx/127.21.66.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":184,"threads_started":1}
W20260620 23:02:21.960026 30968 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:21.954996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:33207 (local address 127.21.66.126:46345)
0620 23:02:21.955052 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:21.955054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:21.955188 (+ 134us) server_negotiation.cc:407] Connection header received
0620 23:02:21.955314 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:21.955315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:21.955328 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:21.955375 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:21.955726 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.956323 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.957090 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:21.957279 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:21.957336 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:21.959928 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.66.66:33207: BlockingRecv error: recv got EOF from 127.21.66.66:33207 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:02:22.009080 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30493
I20260620 23:02:22.013986 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30645
I20260620 23:02:22.018512 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30778
I20260620 23:02:22.023121 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30423
2026-06-20T23:02:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17640 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T23:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:22Z Disabled control of system clock
I20260620 23:02:22.051940 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44785
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:44785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:02:22.142021 30973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:22.142480 30973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:22.142575 30973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:22.145339 30973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:22.145426 30973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:22.145452 30973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:22.145476 30973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:22.145500 30973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:22.148767 30973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:44785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:22.149227 30973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:22.149645 30973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:22.156507 30978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.156590 30981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.156795 30973 server_base.cc:1034] running on GCE node
W20260620 23:02:22.156980 30979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.157308 30973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:22.158020 30973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:22.159189 30973 hybrid_clock.cc:648] HybridClock initialized: now 1781996542159161 us; error 54 us; skew 500 ppm
I20260620 23:02:22.160774 30973 webserver.cc:460] Webserver started at http://127.21.66.126:44743/ using document root <none> and password file <none>
I20260620 23:02:22.161059 30973 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:22.161120 30973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:22.161307 30973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:22.162783 30973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "65e71f10e57f45b1a91fa419a80c2d59"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "77172dcdf35332337877f46d7a1041cb"
server_key_iv: "f7a82ad0fddf38d92d7f5d9f411c6d1f"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.163305 30973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "65e71f10e57f45b1a91fa419a80c2d59"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "77172dcdf35332337877f46d7a1041cb"
server_key_iv: "f7a82ad0fddf38d92d7f5d9f411c6d1f"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.165222 30973 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:02:22.166340 30987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.166620 30973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:22.166730 30973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "65e71f10e57f45b1a91fa419a80c2d59"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "77172dcdf35332337877f46d7a1041cb"
server_key_iv: "f7a82ad0fddf38d92d7f5d9f411c6d1f"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.166831 30973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:22.197908 30973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:22.198347 30973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:22.198515 30973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:22.203002 30973 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:44785
I20260620 23:02:22.203030 31039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:44785 every 8 connection(s)
I20260620 23:02:22.203429 30973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:02:22.204171 31040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:22.207046 31040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59: Bootstrap starting.
I20260620 23:02:22.207453 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 30973
I20260620 23:02:22.207536 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:02:22.207738 21769 external_mini_cluster.cc:1428] Setting key 5d3d07e7d9791819525dde47503a6be1
I20260620 23:02:22.207783 31040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:22.208065 31040 log.cc:824] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:22.208881 31040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59: No bootstrap required, opened a new log
I20260620 23:02:22.210338 31040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e71f10e57f45b1a91fa419a80c2d59" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44785 } }
I20260620 23:02:22.210477 31040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:22.210500 31040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e71f10e57f45b1a91fa419a80c2d59, State: Initialized, Role: FOLLOWER
I20260620 23:02:22.210660 31040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e71f10e57f45b1a91fa419a80c2d59" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44785 } }
I20260620 23:02:22.210985 31040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:22.211059 31040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:22.211138 31040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:22.211886 31040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e71f10e57f45b1a91fa419a80c2d59" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44785 } }
I20260620 23:02:22.212018 31040 leader_election.cc:302] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65e71f10e57f45b1a91fa419a80c2d59; no voters:
I20260620 23:02:22.212208 31040 leader_election.cc:288] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:22.212272 31045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:22.212466 31045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [term 1 LEADER]: Becoming Leader. State: Replica: 65e71f10e57f45b1a91fa419a80c2d59, State: Running, Role: LEADER
I20260620 23:02:22.212512 31040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:22.212566 31045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e71f10e57f45b1a91fa419a80c2d59" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44785 } }
I20260620 23:02:22.212913 31046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65e71f10e57f45b1a91fa419a80c2d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e71f10e57f45b1a91fa419a80c2d59" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44785 } } }
I20260620 23:02:22.213001 31046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:22.213204 31047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e71f10e57f45b1a91fa419a80c2d59. Latest consensus state: current_term: 1 leader_uuid: "65e71f10e57f45b1a91fa419a80c2d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e71f10e57f45b1a91fa419a80c2d59" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44785 } } }
I20260620 23:02:22.213275 31047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:22.213402 31052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:22.214380 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.208296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.21.66.126:44785)
0620 23:02:22.208517 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:22.208530 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:22.208554 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:02:22.209081 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:22.209089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:22.209151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:22.209313 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:22.209492 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.210469 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.211526 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.211979 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.212122 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:22.213422 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:22.213426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:22.213427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:22.213667 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:22.213716 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:22.213735 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:22.213782 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:22.213971 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 23:02:22.214060 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260620 23:02:22.214215 31052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:22.216492 31052 catalog_manager.cc:1347] Generated new cluster ID: af3d467257754ecf936870692a351adc
I20260620 23:02:22.216559 31052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:22.249974 31052 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:22.250821 31052 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:22.259053 31052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65e71f10e57f45b1a91fa419a80c2d59: Generated new TSK 0
I20260620 23:02:22.259299 31052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:22.263697 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44785
--builtin_ntp_servers=127.21.66.84:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 23:02:22.353546 31064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:22.353760 31064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:22.353782 31064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:22.355109 31064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:22.355170 31064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:22.355196 31064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:02:22.356634 31064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:22.356922 31064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:22.357250 31064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:22.362505 31072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.362529 31070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.362582 31069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.362756 31064 server_base.cc:1034] running on GCE node
I20260620 23:02:22.363039 31064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:22.363303 31064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:22.364487 31064 hybrid_clock.cc:648] HybridClock initialized: now 1781996542364463 us; error 44 us; skew 500 ppm
I20260620 23:02:22.365942 31064 webserver.cc:460] Webserver started at http://127.21.66.65:37211/ using document root <none> and password file <none>
I20260620 23:02:22.366267 31064 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:22.366348 31064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:22.366508 31064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:22.367630 31064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "769e11d4965241f381f630e4e7bf93ec"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "8bd7299e54c9065193c0648f3b661c92"
server_key_iv: "463326fb7ee8d16b9a1ad5152b462d76"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.367949 31064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "769e11d4965241f381f630e4e7bf93ec"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "8bd7299e54c9065193c0648f3b661c92"
server_key_iv: "463326fb7ee8d16b9a1ad5152b462d76"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.369256 31064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:02:22.370064 31078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.370247 31064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:22.370342 31064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "769e11d4965241f381f630e4e7bf93ec"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "8bd7299e54c9065193c0648f3b661c92"
server_key_iv: "463326fb7ee8d16b9a1ad5152b462d76"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.370427 31064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:22.390447 31064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:22.390830 31064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:22.391003 31064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:22.391243 31064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:22.391616 31064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:22.391674 31064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.391712 31064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:22.391739 31064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.397904 31064 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:45493
I20260620 23:02:22.397990 31191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:45493 every 8 connection(s)
I20260620 23:02:22.398402 31064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:02:22.399490 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31064
I20260620 23:02:22.399585 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:02:22.399868 21769 external_mini_cluster.cc:1428] Setting key a1fd03b47ee32c7bb9ea4ea5114c36b8
I20260620 23:02:22.400978 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44785
--builtin_ntp_servers=127.21.66.84:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:02:22.405112 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.399460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:38757 (local address 127.21.66.126:44785)
0620 23:02:22.400057 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:22.400061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:22.400074 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:02:22.400473 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:22.400475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:22.400493 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:22.400550 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:22.401379 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.401903 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.403547 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.403741 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.403806 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:22.404787 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:22.404789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:22.404789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:22.404794 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:22.404807 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:22.404811 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:22.404857 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:22.405013 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:02:22.405042 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570}
I20260620 23:02:22.405401 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.399263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44785 (local address 127.21.66.65:38757)
0620 23:02:22.399523 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:22.399578 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:22.400357 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:22.400626 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:22.400630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:22.400643 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:22.401260 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:22.401266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.402019 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.402020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:22.403445 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:22.403448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.403838 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.403840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:22.404021 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:22.404616 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:02:22.404616 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:22.404660 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:02:22.404691 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:22.404878 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:22.404884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:22.404887 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:02:22.404968 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:02:22.405101 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":118,"threads_started":1}
I20260620 23:02:22.406026 31192 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44785
I20260620 23:02:22.406167 31192 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:22.406365 31192 heartbeater.cc:511] Master 127.21.66.126:44785 requested a full tablet report, sending...
I20260620 23:02:22.406903 31004 ts_manager.cc:194] Registered new tserver with Master: 769e11d4965241f381f630e4e7bf93ec (127.21.66.65:45493)
I20260620 23:02:22.407605 31004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.65:38757
W20260620 23:02:22.486210 31195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:22.486475 31195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:22.486511 31195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:22.487707 31195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:22.487784 31195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:22.487836 31195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:02:22.489241 31195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:22.489537 31195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:22.489861 31195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:22.495241 31203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.495428 31201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.495471 31200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.495602 31195 server_base.cc:1034] running on GCE node
I20260620 23:02:22.495901 31195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:22.496205 31195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:22.497408 31195 hybrid_clock.cc:648] HybridClock initialized: now 1781996542497373 us; error 48 us; skew 500 ppm
I20260620 23:02:22.498996 31195 webserver.cc:460] Webserver started at http://127.21.66.66:39385/ using document root <none> and password file <none>
I20260620 23:02:22.499280 31195 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:22.499379 31195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:22.499547 31195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:22.500828 31195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "5a6d0127528f42c88fcd139519bc8c14"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "e964dced69f47e691e7be4052cb5c512"
server_key_iv: "a344da8d317ce740e2f27327fdff2d39"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.501246 31195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "5a6d0127528f42c88fcd139519bc8c14"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "e964dced69f47e691e7be4052cb5c512"
server_key_iv: "a344da8d317ce740e2f27327fdff2d39"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.502897 31195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 23:02:22.503854 31209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.504036 31195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:22.504151 31195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "5a6d0127528f42c88fcd139519bc8c14"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "e964dced69f47e691e7be4052cb5c512"
server_key_iv: "a344da8d317ce740e2f27327fdff2d39"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.504242 31195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:22.522411 31195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:22.522785 31195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:22.522967 31195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:22.523247 31195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:22.523645 31195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:22.523706 31195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.523748 31195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:22.523801 31195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.530146 31195 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:43197
I20260620 23:02:22.530267 31322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:43197 every 8 connection(s)
I20260620 23:02:22.530653 31195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:02:22.536861 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31195
I20260620 23:02:22.536860 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.531519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:35989 (local address 127.21.66.126:44785)
0620 23:02:22.531677 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:22.531680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:22.531931 (+ 251us) server_negotiation.cc:407] Connection header received
0620 23:02:22.532612 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:22.532614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:22.532627 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:22.532699 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:22.533448 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.534021 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.535174 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.535412 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.535512 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:22.536517 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:22.536519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:22.536519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:22.536525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:22.536542 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:22.536547 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:22.536624 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:22.536734 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:02:22.536767 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:02:22.537016 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:02:22.537127 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.531484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44785 (local address 127.21.66.66:35989)
0620 23:02:22.531753 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:22.531804 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:22.532487 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:22.532772 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:22.532777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:22.532788 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:22.533328 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:22.533333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.534171 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.534172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:22.535067 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:22.535069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.535523 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.535524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:22.535676 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:22.536335 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:02:22.536335 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:22.536367 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:02:22.536401 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:22.536626 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:22.536633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:22.536637 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:02:22.536739 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:02:22.536857 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260620 23:02:22.537245 21769 external_mini_cluster.cc:1428] Setting key c34ef6c743de54433451ce2f069fef38
I20260620 23:02:22.537758 31323 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44785
I20260620 23:02:22.537869 31323 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:22.538081 31323 heartbeater.cc:511] Master 127.21.66.126:44785 requested a full tablet report, sending...
I20260620 23:02:22.538563 31004 ts_manager.cc:194] Registered new tserver with Master: 5a6d0127528f42c88fcd139519bc8c14 (127.21.66.66:43197)
I20260620 23:02:22.538625 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44785
--builtin_ntp_servers=127.21.66.84:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:02:22.539055 31004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.66:35989
W20260620 23:02:22.624444 31326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:22.624677 31326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:22.624702 31326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:22.625922 31326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:22.625975 31326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:22.626003 31326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:02:22.627437 31326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:22.627722 31326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:22.628062 31326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:22.633540 31332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.633549 31334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.633608 31326 server_base.cc:1034] running on GCE node
W20260620 23:02:22.633558 31331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.634032 31326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:22.634361 31326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:22.635576 31326 hybrid_clock.cc:648] HybridClock initialized: now 1781996542635543 us; error 47 us; skew 500 ppm
I20260620 23:02:22.637243 31326 webserver.cc:460] Webserver started at http://127.21.66.67:34113/ using document root <none> and password file <none>
I20260620 23:02:22.637521 31326 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:22.637624 31326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:22.637790 31326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:22.639194 31326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "c5fcf05efccf448d8b6c8cfe72ac191f"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "eb4f0299acd3378c8ef7f644feccde2c"
server_key_iv: "36ade7d0612db648c85592558e2e9688"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.639597 31326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "c5fcf05efccf448d8b6c8cfe72ac191f"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "eb4f0299acd3378c8ef7f644feccde2c"
server_key_iv: "36ade7d0612db648c85592558e2e9688"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.641131 31326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:02:22.642004 31340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.642237 31326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:22.642344 31326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "c5fcf05efccf448d8b6c8cfe72ac191f"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "eb4f0299acd3378c8ef7f644feccde2c"
server_key_iv: "36ade7d0612db648c85592558e2e9688"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.642437 31326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:22.655599 31326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:22.655956 31326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:22.656132 31326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:22.656383 31326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:22.656765 31326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:22.656831 31326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.656879 31326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:22.656910 31326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.663359 31326 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:42329
I20260620 23:02:22.663436 31453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:42329 every 8 connection(s)
I20260620 23:02:22.663883 31326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:02:22.670081 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.664788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:54265 (local address 127.21.66.126:44785)
0620 23:02:22.664845 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:22.664847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:22.665117 (+ 270us) server_negotiation.cc:407] Connection header received
0620 23:02:22.665733 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:22.665734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:22.665745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:22.665800 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:22.666730 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.667301 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.668474 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.668698 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.668763 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:22.669800 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:22.669802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:22.669802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:22.669807 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:22.669827 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:22.669832 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:22.669880 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:22.669989 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:02:22.670012 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:02:22.670428 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.664705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44785 (local address 127.21.66.67:54265)
0620 23:02:22.664980 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:22.665018 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:22.665593 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:22.665880 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:22.665884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:22.665893 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:22.666609 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:22.666615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.667425 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.667426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:22.668372 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:22.668375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:22.668812 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:22.668813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:22.668942 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:22.669650 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:02:22.669651 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:22.669683 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:02:22.669710 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:22.669914 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:22.669918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:22.669921 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:02:22.669980 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:02:22.670086 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":69,"threads_started":1}
I20260620 23:02:22.670964 31454 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44785
I20260620 23:02:22.671049 31454 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:22.671195 31454 heartbeater.cc:511] Master 127.21.66.126:44785 requested a full tablet report, sending...
I20260620 23:02:22.671589 31004 ts_manager.cc:194] Registered new tserver with Master: c5fcf05efccf448d8b6c8cfe72ac191f (127.21.66.67:42329)
I20260620 23:02:22.672016 31004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.67:54265
I20260620 23:02:22.674069 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31326
I20260620 23:02:22.674217 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:02:22.674425 21769 external_mini_cluster.cc:1428] Setting key c16528b386f91da6a4dddc6ed4e6f406
I20260620 23:02:22.675571 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:22.677740 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31064
W20260620 23:02:22.677749 31044 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:22.677125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.21.66.126:44785)
0620 23:02:22.677206 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:22.677208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:22.677342 (+ 134us) server_negotiation.cc:407] Connection header received
0620 23:02:22.677391 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:22.677391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:22.677401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:22.677442 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:22.677660 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56580: BlockingRecv error: recv got EOF from 127.0.0.1:56580 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:02:22.682979 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31195
I20260620 23:02:22.688205 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31326
I20260620 23:02:22.692981 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 30973
2026-06-20T23:02:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (669 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T23:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:22Z Disabled control of system clock
I20260620 23:02:22.721982 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45143
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:45143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:02:22.805145 31465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:22.805365 31465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:22.805387 31465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:22.806617 31465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:22.806659 31465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:22.806672 31465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:22.806685 31465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:22.806697 31465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:22.808010 31465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:45143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:22.808211 31465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:22.808451 31465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:22.813539 31473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.813539 31470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.813663 31465 server_base.cc:1034] running on GCE node
W20260620 23:02:22.813625 31471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.814064 31465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:22.814389 31465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:22.815572 31465 hybrid_clock.cc:648] HybridClock initialized: now 1781996542815544 us; error 47 us; skew 500 ppm
I20260620 23:02:22.816845 31465 webserver.cc:460] Webserver started at http://127.21.66.126:40323/ using document root <none> and password file <none>
I20260620 23:02:22.817121 31465 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:22.817194 31465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:22.817344 31465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:22.818559 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "9d186d1391a5bcdf79bb3e0d60ccc737"
server_key_iv: "98e913178cc97de21a59844f732cf2be"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.818928 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "9d186d1391a5bcdf79bb3e0d60ccc737"
server_key_iv: "98e913178cc97de21a59844f732cf2be"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.820247 31465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:02:22.820981 31479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.821156 31465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:02:22.821266 31465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "9d186d1391a5bcdf79bb3e0d60ccc737"
server_key_iv: "98e913178cc97de21a59844f732cf2be"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.821344 31465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:22.824946 31465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:22.825229 31465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:22.825377 31465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:22.829238 31465 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:45143
I20260620 23:02:22.829288 31531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:45143 every 8 connection(s)
I20260620 23:02:22.829645 31465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:02:22.830619 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:22.832981 31532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe: Bootstrap starting.
I20260620 23:02:22.833576 31532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:22.833880 31532 log.cc:824] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:22.834669 31532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe: No bootstrap required, opened a new log
I20260620 23:02:22.836299 31532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45143 } }
I20260620 23:02:22.836441 31532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:22.836463 31532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717c52ed3b0f4a7d8f024646a3f7d7fe, State: Initialized, Role: FOLLOWER
I20260620 23:02:22.836642 31532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45143 } }
I20260620 23:02:22.836735 31532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:22.836776 31532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:22.836771 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31465
I20260620 23:02:22.836843 31532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:22.836875 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:02:22.837069 21769 external_mini_cluster.cc:1428] Setting key b7324739bb8f96f5539114274ae6ed1d
I20260620 23:02:22.837600 31532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45143 } }
I20260620 23:02:22.837709 31532 leader_election.cc:302] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 717c52ed3b0f4a7d8f024646a3f7d7fe; no voters:
I20260620 23:02:22.837875 31532 leader_election.cc:288] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:22.837904 31537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:22.838107 31537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [term 1 LEADER]: Becoming Leader. State: Replica: 717c52ed3b0f4a7d8f024646a3f7d7fe, State: Running, Role: LEADER
I20260620 23:02:22.838181 31532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:22.838217 31537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45143 } }
I20260620 23:02:22.838665 31538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45143 } } }
I20260620 23:02:22.838759 31538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:22.839007 31539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717c52ed3b0f4a7d8f024646a3f7d7fe. Latest consensus state: current_term: 1 leader_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717c52ed3b0f4a7d8f024646a3f7d7fe" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45143 } } }
I20260620 23:02:22.839130 31539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:22.839291 31543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:22.839795 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:22.837673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.21.66.126:45143)
0620 23:02:22.837927 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:22.837937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:22.837976 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:02:22.838107 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:22.838115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:22.838189 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:22.838620 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:22.839167 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:22.839171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:22.839171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:22.839175 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:22.839213 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:22.839228 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:22.839467 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:22.839505 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 23:02:22.839583 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260620 23:02:22.840198 31543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:02:22.841747 31543 catalog_manager.cc:1347] Generated new cluster ID: b8a0cdaf217b4daeaad625d0f53e5946
I20260620 23:02:22.841800 31543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:22.856906 31543 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:22.857128 31543 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:22.865922 31543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 717c52ed3b0f4a7d8f024646a3f7d7fe: Generated new TSK 0
I20260620 23:02:22.866231 31543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:02:22.868019 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45143
--builtin_ntp_servers=127.21.66.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:02:22.973481 31556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:22.973764 31556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:22.973809 31556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:22.975224 31556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:22.975311 31556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:22.975358 31556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:02:22.976828 31556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:22.977141 31556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:22.977489 31556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:22.983669 31564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.983696 31562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:22.984042 31561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:22.984107 31556 server_base.cc:1034] running on GCE node
I20260620 23:02:22.984329 31556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:22.984613 31556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:22.985780 31556 hybrid_clock.cc:648] HybridClock initialized: now 1781996542985767 us; error 26 us; skew 500 ppm
I20260620 23:02:22.987335 31556 webserver.cc:460] Webserver started at http://127.21.66.65:34627/ using document root <none> and password file <none>
I20260620 23:02:22.987619 31556 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:22.987668 31556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:22.987807 31556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:22.988924 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "2878b06b521a4c07b503a55877cc1819"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "b7de3aac2fd0d1a9b323904a5eb90ed6"
server_key_iv: "ab67206e4f8c51f965678ed993030edb"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.989295 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "2878b06b521a4c07b503a55877cc1819"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "b7de3aac2fd0d1a9b323904a5eb90ed6"
server_key_iv: "ab67206e4f8c51f965678ed993030edb"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.990727 31556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:02:22.991518 31570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.991667 31556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:02:22.991772 31556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "2878b06b521a4c07b503a55877cc1819"
format_stamp: "Formatted at 2026-06-20 23:02:22 on dist-test-slave-qgtk"
server_key: "b7de3aac2fd0d1a9b323904a5eb90ed6"
server_key_iv: "ab67206e4f8c51f965678ed993030edb"
server_key_version: "encryptionkey@0"
I20260620 23:02:22.991851 31556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:22.995381 31556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:22.995669 31556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:22.995812 31556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:22.996040 31556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:22.996450 31556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:22.996485 31556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:22.996507 31556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:22.996548 31556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.002759 31556 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:37677
I20260620 23:02:23.002802 31683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:37677 every 8 connection(s)
I20260620 23:02:23.003175 31556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:02:23.004427 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31556
I20260620 23:02:23.004539 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:02:23.004724 21769 external_mini_cluster.cc:1428] Setting key 9df4108605fafb839909ba60749324fc
I20260620 23:02:23.005693 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45143
--builtin_ntp_servers=127.21.66.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:02:23.006906 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:23.004144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:59681 (local address 127.21.66.126:45143)
0620 23:02:23.004254 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:23.004257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:23.004416 (+ 159us) server_negotiation.cc:407] Connection header received
0620 23:02:23.005055 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:23.005056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:23.005064 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:23.005112 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:23.006589 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:23.006590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:23.006591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:23.006597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:23.006614 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:23.006619 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:23.006676 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:23.006817 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:02:23.006834 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 23:02:23.007131 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:23.004038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45143 (local address 127.21.66.65:59681)
0620 23:02:23.004286 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:23.004329 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:23.004944 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:23.005231 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:23.005235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:23.005242 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:23.006413 (+ 1171us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:02:23.006414 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:23.006447 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:02:23.006480 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:23.006672 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:23.006677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:23.006681 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:02:23.006828 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 23:02:23.006895 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":105,"threads_started":1}
I20260620 23:02:23.007718 31684 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45143
I20260620 23:02:23.007813 31684 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:23.007988 31684 heartbeater.cc:511] Master 127.21.66.126:45143 requested a full tablet report, sending...
I20260620 23:02:23.008512 31496 ts_manager.cc:194] Registered new tserver with Master: 2878b06b521a4c07b503a55877cc1819 (127.21.66.65:37677)
W20260620 23:02:23.087989 31687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:23.088253 31687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:23.088289 31687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:23.089481 31687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:23.089555 31687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:23.089598 31687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:02:23.090992 31687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:23.091272 31687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:23.091584 31687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:23.096843 31693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.096921 31692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.096843 31695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:23.097178 31687 server_base.cc:1034] running on GCE node
I20260620 23:02:23.097421 31687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:23.097712 31687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:23.098886 31687 hybrid_clock.cc:648] HybridClock initialized: now 1781996543098867 us; error 33 us; skew 500 ppm
I20260620 23:02:23.100219 31687 webserver.cc:460] Webserver started at http://127.21.66.66:32781/ using document root <none> and password file <none>
I20260620 23:02:23.100484 31687 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:23.100534 31687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:23.100687 31687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:23.101796 31687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "6e0888cf22bd4e6ab920d010ad49c459"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-qgtk"
server_key: "cefcd34d34809505e62297e041f25679"
server_key_iv: "1a759ea8c0cf04418e49e48ef0ebab0d"
server_key_version: "encryptionkey@0"
I20260620 23:02:23.102190 31687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "6e0888cf22bd4e6ab920d010ad49c459"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-qgtk"
server_key: "cefcd34d34809505e62297e041f25679"
server_key_iv: "1a759ea8c0cf04418e49e48ef0ebab0d"
server_key_version: "encryptionkey@0"
I20260620 23:02:23.103593 31687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:23.104470 31701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.104700 31687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:23.104813 31687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "6e0888cf22bd4e6ab920d010ad49c459"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-qgtk"
server_key: "cefcd34d34809505e62297e041f25679"
server_key_iv: "1a759ea8c0cf04418e49e48ef0ebab0d"
server_key_version: "encryptionkey@0"
I20260620 23:02:23.104894 31687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:23.108428 31687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:23.108721 31687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:23.108836 31687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:23.109083 31687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:23.109406 31687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:23.109441 31687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.109462 31687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:23.109475 31687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.115667 31687 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:37785
I20260620 23:02:23.115715 31814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:37785 every 8 connection(s)
I20260620 23:02:23.116058 31687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:02:23.119479 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:23.116842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:55715 (local address 127.21.66.126:45143)
0620 23:02:23.116947 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:23.116950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:23.117182 (+ 232us) server_negotiation.cc:407] Connection header received
0620 23:02:23.117783 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:23.117785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:23.117792 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:23.117868 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:23.119161 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:23.119162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:23.119163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:23.119168 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:23.119192 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:23.119197 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:23.119259 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:23.119391 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 23:02:23.119407 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:02:23.119681 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:23.116827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45143 (local address 127.21.66.66:55715)
0620 23:02:23.117032 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:23.117076 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:23.117655 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:23.117943 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:23.117947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:23.117957 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:23.118977 (+ 1020us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:02:23.118978 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:23.119008 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:02:23.119036 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:23.119270 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:23.119275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:23.119308 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:02:23.119401 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:02:23.119468 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260620 23:02:23.120254 31815 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45143
I20260620 23:02:23.120344 31815 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:23.120456 31815 heartbeater.cc:511] Master 127.21.66.126:45143 requested a full tablet report, sending...
I20260620 23:02:23.120829 31496 ts_manager.cc:194] Registered new tserver with Master: 6e0888cf22bd4e6ab920d010ad49c459 (127.21.66.66:37785)
I20260620 23:02:23.121321 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31687
I20260620 23:02:23.121438 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:02:23.121651 21769 external_mini_cluster.cc:1428] Setting key e4d6f9671eaabf2fcc08bdca6bd87c53
I20260620 23:02:23.122581 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45143
--builtin_ntp_servers=127.21.66.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:02:23.207121 31818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:23.207396 31818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:23.207439 31818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:23.208750 31818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:23.208812 31818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:23.208858 31818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:02:23.210294 31818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:23.210608 31818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:23.210939 31818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:23.216320 31824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.216296 31826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:23.216362 31823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:23.216641 31818 server_base.cc:1034] running on GCE node
I20260620 23:02:23.216897 31818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:23.217125 31818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:23.218297 31818 hybrid_clock.cc:648] HybridClock initialized: now 1781996543218267 us; error 59 us; skew 500 ppm
I20260620 23:02:23.219645 31818 webserver.cc:460] Webserver started at http://127.21.66.67:44577/ using document root <none> and password file <none>
I20260620 23:02:23.219926 31818 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:23.220008 31818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:23.220162 31818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:23.221280 31818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "d085c99802d541798166a560849e22e8"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-qgtk"
server_key: "5b721808870e6cc12a9c5ceb05baca85"
server_key_iv: "b4297f57a7f7e0d199cdaae85edf8211"
server_key_version: "encryptionkey@0"
I20260620 23:02:23.221668 31818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "d085c99802d541798166a560849e22e8"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-qgtk"
server_key: "5b721808870e6cc12a9c5ceb05baca85"
server_key_iv: "b4297f57a7f7e0d199cdaae85edf8211"
server_key_version: "encryptionkey@0"
I20260620 23:02:23.223253 31818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:02:23.224263 31832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.224480 31818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:23.224578 31818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "d085c99802d541798166a560849e22e8"
format_stamp: "Formatted at 2026-06-20 23:02:23 on dist-test-slave-qgtk"
server_key: "5b721808870e6cc12a9c5ceb05baca85"
server_key_iv: "b4297f57a7f7e0d199cdaae85edf8211"
server_key_version: "encryptionkey@0"
I20260620 23:02:23.224675 31818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:23.230015 31818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:23.230412 31818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:23.230561 31818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:23.230782 31818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:23.231164 31818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:23.231212 31818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.231244 31818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:23.231273 31818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:23.238001 31818 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:42505
I20260620 23:02:23.238042 31945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:42505 every 8 connection(s)
I20260620 23:02:23.238489 31818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:02:23.241758 31536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:23.239414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:44539 (local address 127.21.66.126:45143)
0620 23:02:23.239487 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:23.239490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:23.239646 (+ 156us) server_negotiation.cc:407] Connection header received
0620 23:02:23.240135 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:23.240136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:23.240143 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:23.240246 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:23.241539 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:23.241541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:23.241541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:02:23.241546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:23.241562 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:02:23.241566 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:23.241634 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:23.241684 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 23:02:23.241698 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:02:23.241953 31948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:23.239325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45143 (local address 127.21.66.67:44539)
0620 23:02:23.239512 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:23.239554 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:23.240018 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:23.240376 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:23.240380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:23.240389 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:23.241380 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:02:23.241381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:23.241412 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:02:23.241441 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:23.241630 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:23.241633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:23.241635 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:02:23.241684 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:02:23.241733 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260620 23:02:23.242504 31946 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45143
I20260620 23:02:23.242610 31946 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:23.242734 31946 heartbeater.cc:511] Master 127.21.66.126:45143 requested a full tablet report, sending...
I20260620 23:02:23.243074 31496 ts_manager.cc:194] Registered new tserver with Master: d085c99802d541798166a560849e22e8 (127.21.66.67:42505)
I20260620 23:02:23.248131 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31818
I20260620 23:02:23.248225 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:02:23.248414 21769 external_mini_cluster.cc:1428] Setting key 71583222ad2446eb00b676c12f90e0af
I20260620 23:02:23.249576 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:02:23.251451 31536 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:02:23.250962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.21.66.126:45143)
0620 23:02:23.251024 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:23.251026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:23.251124 (+ 98us) server_negotiation.cc:407] Connection header received
0620 23:02:23.251152 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:23.251152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:23.251156 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:23.251209 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:23.251375 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39698: BlockingRecv error: recv got EOF from 127.0.0.1:39698 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:02:23.251490 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31556
I20260620 23:02:23.255846 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31687
I20260620 23:02:23.260187 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31818
I20260620 23:02:23.264824 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31465
2026-06-20T23:02:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (571 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:02:23 dist-test-slave-qgtk krb5kdc[31956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:02:23 dist-test-slave-qgtk krb5kdc[31956](info): set up 2 sockets
Jun 20 23:02:23 dist-test-slave-qgtk krb5kdc[31956](info): commencing operation
krb5kdc: starting...
W20260620 23:02:24.395248 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:02:24 dist-test-slave-qgtk krb5kdc[31956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:02:24.529605 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:36527
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:24.612597 31969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:24.612819 31969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:24.612843 31969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:24.613979 31969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:02:24.614018 31969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:24.614033 31969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:24.614045 31969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:02:24.614058 31969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:02:24.615603 31969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:24.615818 31969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:24.616003 31969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:24.621134 31977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.621127 31974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.621102 31975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:24.621541 31969 server_base.cc:1034] running on GCE node
I20260620 23:02:24.621830 31969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:24.622224 31969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:24.623378 31969 hybrid_clock.cc:648] HybridClock initialized: now 1781996544623363 us; error 36 us; skew 500 ppm
Jun 20 23:02:24 dist-test-slave-qgtk krb5kdc[31956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:24.626103 31969 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:02:24.626695 31969 webserver.cc:460] Webserver started at http://127.21.66.126:45713/ using document root <none> and password file <none>
I20260620 23:02:24.626952 31969 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:24.627002 31969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:24.627142 31969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:24.628228 31969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "e688408adad14defb097e41b8b410655"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-qgtk"
server_key: "6a944a2434a4db85887f5962d27d4366"
server_key_iv: "96424a9583c749ac5975efbb2ec42291"
server_key_version: "encryptionkey@0"
I20260620 23:02:24.628597 31969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "e688408adad14defb097e41b8b410655"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-qgtk"
server_key: "6a944a2434a4db85887f5962d27d4366"
server_key_iv: "96424a9583c749ac5975efbb2ec42291"
server_key_version: "encryptionkey@0"
I20260620 23:02:24.630000 31969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:02:24.630875 31984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.631253 31969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:24.631436 31969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "e688408adad14defb097e41b8b410655"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-qgtk"
server_key: "6a944a2434a4db85887f5962d27d4366"
server_key_iv: "96424a9583c749ac5975efbb2ec42291"
server_key_version: "encryptionkey@0"
I20260620 23:02:24.631534 31969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:24.653494 31969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:24.656425 31969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:24.656584 31969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:24.660735 31969 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:36527
I20260620 23:02:24.660815 32036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:36527 every 8 connection(s)
I20260620 23:02:24.661172 31969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:02:24.662204 32037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:24.664589 32037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655: Bootstrap starting.
I20260620 23:02:24.665223 32037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:24.665277 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 31969
I20260620 23:02:24.665427 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:02:24.665594 32037 log.cc:824] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:24.665657 21769 external_mini_cluster.cc:1428] Setting key 40be600e1e8ef1afa2557348f857694c
I20260620 23:02:24.666533 32037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655: No bootstrap required, opened a new log
I20260620 23:02:24.668874 32037 raft_consensus.cc:348] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e688408adad14defb097e41b8b410655" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 36527 } }
I20260620 23:02:24.669065 32037 raft_consensus.cc:374] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:24.669116 32037 raft_consensus.cc:729] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e688408adad14defb097e41b8b410655, State: Initialized, Role: FOLLOWER
I20260620 23:02:24.669248 32037 consensus_queue.cc:260] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e688408adad14defb097e41b8b410655" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 36527 } }
I20260620 23:02:24.669368 32037 raft_consensus.cc:388] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:02:24.669423 32037 raft_consensus.cc:482] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:02:24.669474 32037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:24.670068 32037 raft_consensus.cc:504] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e688408adad14defb097e41b8b410655" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 36527 } }
I20260620 23:02:24.670218 32037 leader_election.cc:302] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e688408adad14defb097e41b8b410655; no voters:
I20260620 23:02:24.670433 32037 leader_election.cc:288] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:02:24.670481 32042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:24.670652 32042 raft_consensus.cc:686] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [term 1 LEADER]: Becoming Leader. State: Replica: e688408adad14defb097e41b8b410655, State: Running, Role: LEADER
I20260620 23:02:24.670711 32037 sys_catalog.cc:564] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:02:24.670748 32042 consensus_queue.cc:237] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e688408adad14defb097e41b8b410655" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 36527 } }
I20260620 23:02:24.671077 32042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e688408adad14defb097e41b8b410655. Latest consensus state: current_term: 1 leader_uuid: "e688408adad14defb097e41b8b410655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e688408adad14defb097e41b8b410655" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 36527 } } }
I20260620 23:02:24.671185 32042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:24.671082 32044 sys_catalog.cc:455] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e688408adad14defb097e41b8b410655" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e688408adad14defb097e41b8b410655" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 36527 } } }
I20260620 23:02:24.671752 32044 sys_catalog.cc:458] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655 [sys.catalog]: This master's current role is: LEADER
I20260620 23:02:24.672261 32049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:02:24.672911 32049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:02:24 dist-test-slave-qgtk krb5kdc[31956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:02:24.675274 32049 catalog_manager.cc:1347] Generated new cluster ID: a9de356f5a0c43c69c39444f7fc437ae
I20260620 23:02:24.675359 32049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:02:24.685814 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:24.666256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.21.66.126:36527)
0620 23:02:24.666434 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:24.666446 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:24.666496 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:02:24.667078 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:24.667086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:24.667141 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:24.667308 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:24.668009 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:24.668990 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:24.670043 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:24.670849 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:24.671338 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:24.674226 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:24.674232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:24.674232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:24.674247 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:24.683436 (+ 9189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:24.684092 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:24.684096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:24.684097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:24.684235 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:24.684510 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:24.684511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:24.684511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:24.684881 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:24.685029 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:24.685228 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 23:02:24.685541 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260620 23:02:24.692958 32049 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:02:24.693779 32049 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:02:24.705997 32049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655: Generated new TSK 0
I20260620 23:02:24.706296 32049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:02:24.747193 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:36527
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:24.837294 32063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:24.837520 32063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:24.837548 32063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:24.838882 32063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:24.838929 32063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:24.838958 32063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:02:24.840503 32063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:24.840766 32063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:24.841037 32063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:24.846411 32071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.846460 32069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:24.846441 32068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:24.846550 32063 server_base.cc:1034] running on GCE node
I20260620 23:02:24.846897 32063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:24.847208 32063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:24.848412 32063 hybrid_clock.cc:648] HybridClock initialized: now 1781996544848387 us; error 40 us; skew 500 ppm
Jun 20 23:02:24 dist-test-slave-qgtk krb5kdc[31956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:24.851334 32063 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:02:24.851862 32063 webserver.cc:460] Webserver started at http://127.21.66.65:36745/ using document root <none> and password file <none>
I20260620 23:02:24.852104 32063 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:24.852173 32063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:24.852326 32063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:24.853464 32063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "3a3b61e259bf4118a67cd354ac14d59a"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-qgtk"
server_key: "03375be06401511aecf3c323bc1f148b"
server_key_iv: "68ee3ab22e8b9ab93a308c2f8a440e26"
server_key_version: "encryptionkey@0"
I20260620 23:02:24.853835 32063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "3a3b61e259bf4118a67cd354ac14d59a"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-qgtk"
server_key: "03375be06401511aecf3c323bc1f148b"
server_key_iv: "68ee3ab22e8b9ab93a308c2f8a440e26"
server_key_version: "encryptionkey@0"
I20260620 23:02:24.855424 32063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:02:24.856431 32078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.856637 32063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:02:24.856760 32063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "3a3b61e259bf4118a67cd354ac14d59a"
format_stamp: "Formatted at 2026-06-20 23:02:24 on dist-test-slave-qgtk"
server_key: "03375be06401511aecf3c323bc1f148b"
server_key_iv: "68ee3ab22e8b9ab93a308c2f8a440e26"
server_key_version: "encryptionkey@0"
I20260620 23:02:24.856851 32063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:24.880800 32063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:24.883422 32063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:24.883610 32063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:24.883896 32063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:24.884310 32063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:24.884364 32063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.884408 32063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:24.884435 32063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:24.890579 32063 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:36357
I20260620 23:02:24.890626 32191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:36357 every 8 connection(s)
I20260620 23:02:24.890973 32063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:02:24.893127 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 32063
I20260620 23:02:24.893265 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:02:24.893476 21769 external_mini_cluster.cc:1428] Setting key 291d71ca4e2b7b30c6d9e90996353ea1
Jun 20 23:02:24 dist-test-slave-qgtk krb5kdc[31956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996544, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:02:24.902727 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:24.891758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:59263 (local address 127.21.66.126:36527)
0620 23:02:24.891805 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:24.891807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:24.892064 (+ 257us) server_negotiation.cc:407] Connection header received
0620 23:02:24.892648 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:24.892650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:24.892662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:24.892717 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:24.894044 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:24.894664 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:24.896099 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:24.896454 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:24.896533 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:24.899273 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:24.899276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:24.899276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:24.899287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:24.901097 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:24.901544 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:24.901545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:24.901546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:24.901619 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:24.901896 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:24.901897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:24.901897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:24.902091 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:24.902207 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:24.902492 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:02:24.902599 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:02:24.902931 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:24.891739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:36527 (local address 127.21.66.65:59263)
0620 23:02:24.891928 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:24.891967 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:24.892522 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:24.892794 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:24.892798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:24.893660 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:24.893949 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:24.893956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:24.894766 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:24.894767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:24.896003 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:24.896006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:24.896543 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:24.896544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:24.896650 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:24.897320 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:24.897330 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:24.899144 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:24.901191 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:24.901203 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:24.901206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:24.901457 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:24.901707 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:24.901708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:24.901708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:24.901823 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:24.902212 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:24.902218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:24.902355 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 23:02:24.902448 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:02:24.902582 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260620 23:02:24.903510 32192 heartbeater.cc:347] Connected to a master server at 127.21.66.126:36527
I20260620 23:02:24.903714 32192 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:24.903934 32192 heartbeater.cc:511] Master 127.21.66.126:36527 requested a full tablet report, sending...
I20260620 23:02:24.904547 32001 ts_manager.cc:194] Registered new tserver with Master: 3a3b61e259bf4118a67cd354ac14d59a (127.21.66.65:36357)
I20260620 23:02:24.905174 32001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:59263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:02:24.922290 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:36527
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:25.010095 32197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:25.010344 32197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:25.010371 32197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:25.011627 32197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:25.011677 32197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:25.011708 32197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:02:25.013211 32197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:25.013480 32197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:25.013741 32197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:25.019274 32203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:25.019300 32205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:25.019618 32197 server_base.cc:1034] running on GCE node
W20260620 23:02:25.019307 32202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:25.019943 32197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:25.020201 32197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:25.021395 32197 hybrid_clock.cc:648] HybridClock initialized: now 1781996545021381 us; error 31 us; skew 500 ppm
Jun 20 23:02:25 dist-test-slave-qgtk krb5kdc[31956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:25.024194 32197 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:02:25.024706 32197 webserver.cc:460] Webserver started at http://127.21.66.66:39481/ using document root <none> and password file <none>
I20260620 23:02:25.024958 32197 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:25.025039 32197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:25.025194 32197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:25.026394 32197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "6dba8fdc2de34f279d30ceb057943691"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-qgtk"
server_key: "b9df8e082f07e24ad91a541e0640bec1"
server_key_iv: "b85d104493583f366f319cb533ee3666"
server_key_version: "encryptionkey@0"
I20260620 23:02:25.026782 32197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "6dba8fdc2de34f279d30ceb057943691"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-qgtk"
server_key: "b9df8e082f07e24ad91a541e0640bec1"
server_key_iv: "b85d104493583f366f319cb533ee3666"
server_key_version: "encryptionkey@0"
I20260620 23:02:25.028255 32197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:02:25.029078 32212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.029289 32197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:25.029418 32197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "6dba8fdc2de34f279d30ceb057943691"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-qgtk"
server_key: "b9df8e082f07e24ad91a541e0640bec1"
server_key_iv: "b85d104493583f366f319cb533ee3666"
server_key_version: "encryptionkey@0"
I20260620 23:02:25.029513 32197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:25.044478 32197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:25.047199 32197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:25.047398 32197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:25.047693 32197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:25.048054 32197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:25.048116 32197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.048167 32197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:25.048223 32197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.055210 32197 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:40917
I20260620 23:02:25.055269 32325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:40917 every 8 connection(s)
I20260620 23:02:25.055680 32197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:02:25.058104 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 32197
I20260620 23:02:25.058296 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:02:25.058490 21769 external_mini_cluster.cc:1428] Setting key 93f5a422052dc860f3307e342c6a94eb
Jun 20 23:02:25 dist-test-slave-qgtk krb5kdc[31956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
I20260620 23:02:25.068015 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.056564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:41163 (local address 127.21.66.126:36527)
0620 23:02:25.056637 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.056639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.056920 (+ 281us) server_negotiation.cc:407] Connection header received
0620 23:02:25.057454 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.057455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.057475 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.057543 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:25.058874 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.059752 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.061478 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.061676 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.061742 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.064307 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:25.064309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:25.064310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:25.064319 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:25.066248 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:25.066778 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:25.066780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:25.066780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:25.066853 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:25.067176 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:25.067177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:25.067178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:25.067368 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:25.067467 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.067802 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.067902 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:02:25.068274 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.056557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:36527 (local address 127.21.66.66:41163)
0620 23:02:25.056743 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.056792 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:25.057342 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:25.057606 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:25.057610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:25.058387 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:25.058700 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.058707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.059971 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.059972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.061363 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.061367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.061776 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.061778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.061943 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.062592 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:25.062601 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:25.064190 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:25.066367 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:25.066376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:25.066378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:25.066665 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:25.066937 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:25.066938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:25.066939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:25.067082 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:25.067456 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:25.067464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:25.067636 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:02:25.067776 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:02:25.067971 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260620 23:02:25.069109 32326 heartbeater.cc:347] Connected to a master server at 127.21.66.126:36527
I20260620 23:02:25.069320 32326 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:25.069540 32326 heartbeater.cc:511] Master 127.21.66.126:36527 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
I20260620 23:02:25.070220 32001 ts_manager.cc:194] Registered new tserver with Master: 6dba8fdc2de34f279d30ceb057943691 (127.21.66.66:40917)
I20260620 23:02:25.070648 32001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:41163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:02:25.090204 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:36527
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:02:25.179456 32331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:02:25.179713 32331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:02:25.179742 32331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:02:25.181180 32331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:02:25.181247 32331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:02:25.181279 32331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:02:25.183133 32331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:02:25.183473 32331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:02:25.183773 32331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:02:25.189913 32336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:02:25.189949 32339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:25.190075 32331 server_base.cc:1034] running on GCE node
W20260620 23:02:25.189908 32337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:02:25.190544 32331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:02:25.190886 32331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:02:25.192099 32331 hybrid_clock.cc:648] HybridClock initialized: now 1781996545192084 us; error 38 us; skew 500 ppm
Jun 20 23:02:25 dist-test-slave-qgtk krb5kdc[31956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:02:25.195252 32331 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:02:25.195786 32331 webserver.cc:460] Webserver started at http://127.21.66.67:45281/ using document root <none> and password file <none>
I20260620 23:02:25.196064 32331 fs_manager.cc:359] Metadata directory not provided
I20260620 23:02:25.196141 32331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:02:25.196321 32331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:02:25.197556 32331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "d1adf3f3da7641f296a301c436f03ba2"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-qgtk"
server_key: "565e64fee25deaa61a7fd6c3336d5437"
server_key_iv: "9a7ce2ee12b173ef080802ddb262682a"
server_key_version: "encryptionkey@0"
I20260620 23:02:25.197959 32331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "d1adf3f3da7641f296a301c436f03ba2"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-qgtk"
server_key: "565e64fee25deaa61a7fd6c3336d5437"
server_key_iv: "9a7ce2ee12b173ef080802ddb262682a"
server_key_version: "encryptionkey@0"
I20260620 23:02:25.199554 32331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:25.200500 32346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.200745 32331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:02:25.200881 32331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "d1adf3f3da7641f296a301c436f03ba2"
format_stamp: "Formatted at 2026-06-20 23:02:25 on dist-test-slave-qgtk"
server_key: "565e64fee25deaa61a7fd6c3336d5437"
server_key_iv: "9a7ce2ee12b173ef080802ddb262682a"
server_key_version: "encryptionkey@0"
I20260620 23:02:25.200968 32331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:02:25.257841 32331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:02:25.261004 32331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:02:25.261159 32331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:02:25.261447 32331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:02:25.261792 32331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:02:25.261828 32331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.261850 32331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:02:25.261863 32331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:02:25.268334 32331 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:45847
I20260620 23:02:25.268409 32459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:45847 every 8 connection(s)
I20260620 23:02:25.268764 32331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:02:25 dist-test-slave-qgtk krb5kdc[31956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996545, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:02:25.277240 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 32331
I20260620 23:02:25.277354 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:02:25.277551 21769 external_mini_cluster.cc:1428] Setting key 7c744ed4c877c08c3055fce919477e1d
I20260620 23:02:25.280629 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.269681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:47963 (local address 127.21.66.126:36527)
0620 23:02:25.269807 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.269809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.269986 (+ 177us) server_negotiation.cc:407] Connection header received
0620 23:02:25.270672 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.270674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.270710 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.270873 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:25.272110 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.272841 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.274063 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.274329 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.274402 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.277091 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:25.277092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:25.277093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:25.277103 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:25.279058 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:25.279541 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:25.279542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:25.279542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:25.279604 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:25.279908 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:25.279908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:25.279909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:25.280035 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:25.280092 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.280473 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.280553 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 23:02:25.280905 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.269664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:36527 (local address 127.21.66.67:47963)
0620 23:02:25.269819 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.269878 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:25.270492 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:25.270994 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:25.271006 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:25.271783 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:02:25.272006 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.272012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.272980 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.272981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.273962 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.273965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.274447 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.274448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.274576 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.275239 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:02:25.275248 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:02:25.276944 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:02:25.279144 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:25.279152 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:25.279155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:25.279437 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:25.279676 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:02:25.279677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:02:25.279677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:02:25.279842 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:02:25.280139 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:02:25.280145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:02:25.280288 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 23:02:25.280407 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:02:25.280552 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260620 23:02:25.281636 32460 heartbeater.cc:347] Connected to a master server at 127.21.66.126:36527
I20260620 23:02:25.281730 32460 heartbeater.cc:464] Registering TS with master...
I20260620 23:02:25.281952 32460 heartbeater.cc:511] Master 127.21.66.126:36527 requested a full tablet report, sending...
I20260620 23:02:25.282539 32001 ts_manager.cc:194] Registered new tserver with Master: d1adf3f3da7641f296a301c436f03ba2 (127.21.66.67:45847)
I20260620 23:02:25.282963 32001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:47963
I20260620 23:02:25.289695 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:02:25.299021 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.291445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.21.66.126:36527)
0620 23:02:25.291541 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.291544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.291649 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:02:25.291848 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.291850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.291861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.291943 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:02:25.292596 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.293150 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.294021 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.294352 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.294447 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.295919 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:02:25.295921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:02:25.295922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:02:25.295930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:02:25.297440 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:25.297976 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:25.297978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:25.297979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:25.298046 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:02:25.298348 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:02:25.298349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:02:25.298350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:02:25.298531 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:02:25.298654 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.298795 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.298892 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:02:25.301062 32001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:02:25.302992 32001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:02:25.310075 32471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.305765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:40917 (local address 127.0.0.1:48844)
0620 23:02:25.305927 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.305972 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:25.306041 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:25.306525 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:25.306528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:25.306530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:25.306616 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.306619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.307678 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.307679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.308794 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.308797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.309731 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.309732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.309817 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.309830 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:02:25.309934 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:02:25.309990 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260620 23:02:25.310082 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.306035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.21.66.66:40917)
0620 23:02:25.306362 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.306366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.306378 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:02:25.306421 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.306423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.306441 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.306522 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:25.306706 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.307574 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.308896 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.309636 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.309724 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.309789 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.309953 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.309993 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":123,"threads_started":1}
I20260620 23:02:25.310421 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.305903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:36357 (local address 127.0.0.1:51830)
0620 23:02:25.306324 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.306335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:25.306386 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:25.306773 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:25.306774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:25.306777 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:25.306836 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.306838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.308160 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.308161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.309153 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.309155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.310173 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.310175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.310246 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.310251 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:02:25.310299 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:02:25.310342 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":26,"thread_start_us":100,"threads_started":1}
I20260620 23:02:25.310592 32472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.306183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:45847 (local address 127.0.0.1:34642)
0620 23:02:25.306619 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.306629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:25.306674 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:25.306944 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:25.306945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:25.306947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:25.306984 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.306985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.308263 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.308264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.309342 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.309345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.310368 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.310369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.310424 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.310428 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:02:25.310476 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 23:02:25.310518 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":72,"threads_started":1}
I20260620 23:02:25.310740 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.306033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.21.66.65:36357)
0620 23:02:25.306320 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.306325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.306535 (+ 210us) server_negotiation.cc:407] Connection header received
0620 23:02:25.306583 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.306586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.306602 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.306689 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:25.306918 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.308044 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.309239 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.309985 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.310057 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.310111 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.310599 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.310657 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":124,"threads_started":1}
I20260620 23:02:25.311048 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.306367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.21.66.67:45847)
0620 23:02:25.306535 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.306539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.306736 (+ 197us) server_negotiation.cc:407] Connection header received
0620 23:02:25.306783 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.306786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.306801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.306868 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:25.307067 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.308151 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.309431 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.310195 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.310257 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.310330 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.310720 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.310770 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260620 23:02:25.311220 32260 tablet_service.cc:1467] Processing CreateTablet for tablet 947a0146bb7e4f6b9b227b493fbaefce (DEFAULT_TABLE table=test-table [id=28b6aa461fe945929f68cdabddc2972a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:25.311388 32126 tablet_service.cc:1467] Processing CreateTablet for tablet 947a0146bb7e4f6b9b227b493fbaefce (DEFAULT_TABLE table=test-table [id=28b6aa461fe945929f68cdabddc2972a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:25.311642 32260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a0146bb7e4f6b9b227b493fbaefce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:25.311602 32394 tablet_service.cc:1467] Processing CreateTablet for tablet 947a0146bb7e4f6b9b227b493fbaefce (DEFAULT_TABLE table=test-table [id=28b6aa461fe945929f68cdabddc2972a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:02:25.311734 32126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a0146bb7e4f6b9b227b493fbaefce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:25.311887 32394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 947a0146bb7e4f6b9b227b493fbaefce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:02:25.314502 32477 tablet_bootstrap.cc:492] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Bootstrap starting.
I20260620 23:02:25.315112 32479 tablet_bootstrap.cc:492] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a: Bootstrap starting.
I20260620 23:02:25.315155 32477 tablet_bootstrap.cc:654] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:25.315775 32477 log.cc:824] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:25.315927 32479 tablet_bootstrap.cc:654] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:25.316479 32479 log.cc:824] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:25.316671 32477 tablet_bootstrap.cc:492] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: No bootstrap required, opened a new log
I20260620 23:02:25.316751 32477 ts_tablet_manager.cc:1397] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:25.316869 32478 tablet_bootstrap.cc:492] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Bootstrap starting.
I20260620 23:02:25.317246 32479 tablet_bootstrap.cc:492] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a: No bootstrap required, opened a new log
I20260620 23:02:25.317369 32479 ts_tablet_manager.cc:1397] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:02:25.317517 32478 tablet_bootstrap.cc:654] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Neither blocks nor log segments found. Creating new log.
I20260620 23:02:25.318053 32478 log.cc:824] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:02:25.318882 32477 raft_consensus.cc:348] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.319020 32477 raft_consensus.cc:374] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:25.319044 32477 raft_consensus.cc:729] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dba8fdc2de34f279d30ceb057943691, State: Initialized, Role: FOLLOWER
I20260620 23:02:25.319167 32477 consensus_queue.cc:260] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.319201 32479 raft_consensus.cc:348] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.319393 32479 raft_consensus.cc:374] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:25.319438 32479 raft_consensus.cc:729] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3b61e259bf4118a67cd354ac14d59a, State: Initialized, Role: FOLLOWER
I20260620 23:02:25.319459 32326 heartbeater.cc:503] Master 127.21.66.126:36527 was elected leader, sending a full tablet report...
I20260620 23:02:25.319559 32479 consensus_queue.cc:260] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.319705 32477 ts_tablet_manager.cc:1428] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:02:25.319852 32479 ts_tablet_manager.cc:1428] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:02:25.319967 32192 heartbeater.cc:503] Master 127.21.66.126:36527 was elected leader, sending a full tablet report...
I20260620 23:02:25.320468 32478 tablet_bootstrap.cc:492] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: No bootstrap required, opened a new log
I20260620 23:02:25.320580 32478 ts_tablet_manager.cc:1397] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:02:25.322369 32478 raft_consensus.cc:348] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.322527 32478 raft_consensus.cc:374] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:02:25.322571 32478 raft_consensus.cc:729] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1adf3f3da7641f296a301c436f03ba2, State: Initialized, Role: FOLLOWER
I20260620 23:02:25.322691 32478 consensus_queue.cc:260] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.322994 32478 ts_tablet_manager.cc:1428] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:02:25.323004 32460 heartbeater.cc:503] Master 127.21.66.126:36527 was elected leader, sending a full tablet report...
I20260620 23:02:25.340965 32484 raft_consensus.cc:482] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:02:25.341136 32484 raft_consensus.cc:504] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.341589 32484 leader_election.cc:288] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1adf3f3da7641f296a301c436f03ba2 (127.21.66.67:45847), 6dba8fdc2de34f279d30ceb057943691 (127.21.66.66:40917)
I20260620 23:02:25.346213 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.341961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:48493 (local address 127.21.66.66:40917)
0620 23:02:25.342009 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.342011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.342250 (+ 239us) server_negotiation.cc:407] Connection header received
0620 23:02:25.342282 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.342283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.342299 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.342344 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:25.342520 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.343225 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.345085 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.345719 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.345819 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.345828 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.346068 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.346143 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:02:25.346213 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.341961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:47813 (local address 127.21.66.67:45847)
0620 23:02:25.342021 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.342023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.342031 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:02:25.342061 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.342062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.342071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.342216 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:02:25.342443 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.343273 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.345084 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.345736 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.345819 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.345828 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.346068 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.346136 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 23:02:25.346237 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.341852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:40917 (local address 127.21.66.65:48493)
0620 23:02:25.342103 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.342164 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:25.342222 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:25.342376 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:25.342376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:25.342378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:25.342443 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.342444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.343367 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.343368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.344936 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.344938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.345873 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.345874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.345966 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.345971 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:02:25.346042 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:02:25.346098 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":173,"thread_start_us":127,"threads_started":1}
I20260620 23:02:25.346537 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.341839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:45847 (local address 127.21.66.65:47813)
0620 23:02:25.341898 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.341919 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:02:25.341965 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:02:25.342232 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:02:25.342233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:02:25.342235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:02:25.342321 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.342322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.343366 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.343368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.344876 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:02:25.344879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.345873 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.345874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:02:25.345971 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.345972 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:02:25.346042 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 23:02:25.346098 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":119}
I20260620 23:02:25.346877 32414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a0146bb7e4f6b9b227b493fbaefce" candidate_uuid: "3a3b61e259bf4118a67cd354ac14d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1adf3f3da7641f296a301c436f03ba2" is_pre_election: true
I20260620 23:02:25.347078 32414 raft_consensus.cc:2454] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3b61e259bf4118a67cd354ac14d59a in term 0.
I20260620 23:02:25.347030 32280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a0146bb7e4f6b9b227b493fbaefce" candidate_uuid: "3a3b61e259bf4118a67cd354ac14d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dba8fdc2de34f279d30ceb057943691" is_pre_election: true
I20260620 23:02:25.347208 32280 raft_consensus.cc:2454] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3b61e259bf4118a67cd354ac14d59a in term 0.
I20260620 23:02:25.347330 32080 leader_election.cc:302] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3b61e259bf4118a67cd354ac14d59a, d1adf3f3da7641f296a301c436f03ba2; no voters:
I20260620 23:02:25.347500 32484 raft_consensus.cc:2781] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:02:25.347558 32484 raft_consensus.cc:482] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:02:25.347592 32484 raft_consensus.cc:3037] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:25.348335 32484 raft_consensus.cc:504] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.348486 32484 leader_election.cc:288] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [CANDIDATE]: Term 1 election: Requested vote from peers d1adf3f3da7641f296a301c436f03ba2 (127.21.66.67:45847), 6dba8fdc2de34f279d30ceb057943691 (127.21.66.66:40917)
I20260620 23:02:25.348634 32414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a0146bb7e4f6b9b227b493fbaefce" candidate_uuid: "3a3b61e259bf4118a67cd354ac14d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1adf3f3da7641f296a301c436f03ba2"
I20260620 23:02:25.348637 32280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "947a0146bb7e4f6b9b227b493fbaefce" candidate_uuid: "3a3b61e259bf4118a67cd354ac14d59a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dba8fdc2de34f279d30ceb057943691"
I20260620 23:02:25.348737 32414 raft_consensus.cc:3037] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:25.348737 32280 raft_consensus.cc:3037] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:02:25.349534 32414 raft_consensus.cc:2454] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3b61e259bf4118a67cd354ac14d59a in term 1.
I20260620 23:02:25.349709 32280 raft_consensus.cc:2454] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3b61e259bf4118a67cd354ac14d59a in term 1.
I20260620 23:02:25.349752 32080 leader_election.cc:302] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3b61e259bf4118a67cd354ac14d59a, d1adf3f3da7641f296a301c436f03ba2; no voters:
I20260620 23:02:25.349857 32484 raft_consensus.cc:2781] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:02:25.350006 32484 raft_consensus.cc:686] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [term 1 LEADER]: Becoming Leader. State: Replica: 3a3b61e259bf4118a67cd354ac14d59a, State: Running, Role: LEADER
I20260620 23:02:25.350081 32484 consensus_queue.cc:237] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } }
I20260620 23:02:25.351070 32001 catalog_manager.cc:5495] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3b61e259bf4118a67cd354ac14d59a (127.21.66.65). New cstate: current_term: 1 leader_uuid: "3a3b61e259bf4118a67cd354ac14d59a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3b61e259bf4118a67cd354ac14d59a" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 36357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 } health_report { overall_health: UNKNOWN } } }
I20260620 23:02:25.361853 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.359150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.21.66.65:36357)
0620 23:02:25.359225 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:02:25.359227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:02:25.359288 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:02:25.359375 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:02:25.359376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:02:25.359400 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:02:25.359467 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:02:25.359613 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.360142 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.361061 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:02:25.361278 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:02:25.361344 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:02:25.361488 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:02:25.361597 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:02:25.361660 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:02:25.361697 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 23:02:25.361717 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:02:25.363827 32280 raft_consensus.cc:1261] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 1 FOLLOWER]: Refusing update from remote peer 3a3b61e259bf4118a67cd354ac14d59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:25.363828 32414 raft_consensus.cc:1261] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 1 FOLLOWER]: Refusing update from remote peer 3a3b61e259bf4118a67cd354ac14d59a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:02:25.364261 32487 consensus_queue.cc:1035] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dba8fdc2de34f279d30ceb057943691" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 40917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:25.364420 32484 consensus_queue.cc:1035] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1adf3f3da7641f296a301c436f03ba2" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:02:25.367579 32491 mvcc.cc:205] Tried to move back new op lower bound from 7299057849806061568 to 7299057849753694208. Current Snapshot: MvccSnapshot[applied={T|T < 7299057849806061568}]
I20260620 23:02:25.367781 32492 mvcc.cc:205] Tried to move back new op lower bound from 7299057849806061568 to 7299057849753694208. Current Snapshot: MvccSnapshot[applied={T|T < 7299057849806061568}]
I20260620 23:02:25.369489 32000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:25.369609 32000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:02:25.370489 32000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e688408adad14defb097e41b8b410655: Sending DeleteTablet for 3 replicas of tablet 947a0146bb7e4f6b9b227b493fbaefce
I20260620 23:02:25.370886 32126 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a0146bb7e4f6b9b227b493fbaefce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:25 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:51830
I20260620 23:02:25.370939 32260 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a0146bb7e4f6b9b227b493fbaefce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:25 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:48844
I20260620 23:02:25.370904 32394 tablet_service.cc:1514] Processing DeleteTablet for tablet 947a0146bb7e4f6b9b227b493fbaefce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:02:25 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:34642
I20260620 23:02:25.371243 32499 tablet_replica.cc:330] T 947a0146bb7e4f6b9b227b493fbaefce P 3a3b61e259bf4118a67cd354ac14d59a: stopping tablet replica
I20260620 23:02:25.371409 32501 tablet_replica.cc:330] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: stopping tablet replica
I20260620 23:02:25.371430 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 32063
I20260620 23:02:25.372241 32501 raft_consensus.cc:2229] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:25.372434 32501 raft_consensus.cc:2258] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:25.372982 32501 ts_tablet_manager.cc:1905] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:25.373404 32500 tablet_replica.cc:330] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: stopping tablet replica
I20260620 23:02:25.373562 32500 raft_consensus.cc:2229] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:02:25.373709 32500 raft_consensus.cc:2258] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:02:25.374418 32500 ts_tablet_manager.cc:1905] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:02:25.376855 32501 ts_tablet_manager.cc:1918] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:25.376978 32501 log.cc:1196] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/947a0146bb7e4f6b9b227b493fbaefce
I20260620 23:02:25.376992 32500 ts_tablet_manager.cc:1918] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:02:25.377058 32500 log.cc:1196] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/947a0146bb7e4f6b9b227b493fbaefce
I20260620 23:02:25.377300 32501 ts_tablet_manager.cc:1939] T 947a0146bb7e4f6b9b227b493fbaefce P d1adf3f3da7641f296a301c436f03ba2: Deleting consensus metadata
I20260620 23:02:25.377332 32500 ts_tablet_manager.cc:1939] T 947a0146bb7e4f6b9b227b493fbaefce P 6dba8fdc2de34f279d30ceb057943691: Deleting consensus metadata
I20260620 23:02:25.377837 31986 catalog_manager.cc:4841] TS d1adf3f3da7641f296a301c436f03ba2 (127.21.66.67:45847): tablet 947a0146bb7e4f6b9b227b493fbaefce (table test-table [id=28b6aa461fe945929f68cdabddc2972a]) successfully deleted
I20260620 23:02:25.377837 31985 catalog_manager.cc:4841] TS 6dba8fdc2de34f279d30ceb057943691 (127.21.66.66:40917): tablet 947a0146bb7e4f6b9b227b493fbaefce (table test-table [id=28b6aa461fe945929f68cdabddc2972a]) successfully deleted
I20260620 23:02:25.378037 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 32197
W20260620 23:02:25.378059 31986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:36357 (error 108)
I20260620 23:02:25.379030 32472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:02:25.378861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:36357 (local address 127.0.0.1:51834)
0620 23:02:25.378934 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 23:02:25.378985 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:36357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 23:02:25.379208 31986 catalog_manager.cc:4584] TS 3a3b61e259bf4118a67cd354ac14d59a (127.21.66.65:36357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 947a0146bb7e4f6b9b227b493fbaefce: Network error: Client connection negotiation failed: client connection to 127.21.66.65:36357: connect: Connection refused (error 111)
I20260620 23:02:25.384171 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 32331
I20260620 23:02:25.388934 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 31969
2026-06-20T23:02:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2124 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:02:25 dist-test-slave-qgtk krb5kdc[32505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:02:25 dist-test-slave-qgtk krb5kdc[32505](info): set up 2 sockets
Jun 20 23:02:25 dist-test-slave-qgtk krb5kdc[32505](info): commencing operation
krb5kdc: starting...
W20260620 23:02:26.479624 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:02:26 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:02:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/rangeradmin_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/rangeradmin_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/rangerlookup_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/rangerlookup_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
I20260620 23:02:26.669926 21769 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxQfmfq/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/postgres -l logfile start
2026-06-20 23:02:29.449 UTC [32535] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:02:29.449 UTC [32535] LOG: listening on IPv4 address "127.21.66.84", port 49851
2026-06-20 23:02:29.454 UTC [32535] LOG: listening on Unix socket "/tmp/.s.PGSQL.49851"
2026-06-20 23:02:29.491 UTC [32538] LOG: database system was shut down at 2026-06-20 23:02:27 UTC
2026-06-20 23:02:29.496 UTC [32535] LOG: database system is ready to accept connections
I20260620 23:02:30.471168 21769 mini_postgres.cc:96] Postgres bound to 49851
2026-06-20 23:02:30.477 UTC [32546] FATAL: database "slave" does not exist
127.21.66.84:49851 - accepting connections
I20260620 23:02:30.477957 21769 mini_ranger.cc:161] Starting Ranger...
I20260620 23:02:30.496899 21769 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:02:31.082302 21769 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:02:31.082398 21769 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-admin
2026-06-20 23:02:31,375 [I] DB FLAVOR :POSTGRES
2026-06-20 23:02:31,375 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:02:31,375 [I] Checking connection..
2026-06-20 23:02:31,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:02:31,628 [I] Checking connection passed.
2026-06-20 23:02:31,629 [I] --------- Verifying version history table ---------
2026-06-20 23:02:31,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:02:31,901 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:02:31,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:02:32,181 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:02:32,182 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:02:32,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:02:32.490 UTC [32610] WARNING: there is no transaction in progress
2026-06-20 23:02:32,497 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:02:32,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:02:32,758 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:02:32,758 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:02:32,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:02:33,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:02:33,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qgtk.c.gcp-upstream.internal','N') ;"
2026-06-20 23:02:33,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:02:33,845 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:02:33,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:02:34,119 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:02:34,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:02:34,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:02:34,681 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:02:34,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:02:35.823 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:35.842 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:35.862 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:35.880 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:35.898 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:35.919 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:35.991 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.000 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.018 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.030 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.042 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.058 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.070 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.077 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.085 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36.093 UTC [32718] WARNING: there is no transaction in progress
2026-06-20 23:02:36,901 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:02:36,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:02:37,170 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:02:37,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:02:37,445 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:02:37,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:02:37,714 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:02:37,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:02:37,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:02:38,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qgtk.c.gcp-upstream.internal';"
2026-06-20 23:02:38,497 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:02:38,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:02:38,766 [I] DB_PATCHES have already been applied
I20260620 23:02:38.772004 21769 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxQfmfq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:02:38.772076 21769 mini_ranger.cc:193] Using host: 127.21.66.84
I20260620 23:02:38.775439 21769 mini_ranger.cc:227] Ranger admin URL: http://127.21.66.84:43113
Jun 20, 2026 11:02:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:02:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:02:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:02:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:02:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:02:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:02:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:02:39.903888 21769 mini_ranger.cc:160] Time spent starting Ranger: real 9.426s user 0.005s sys 0.005s
Jun 20, 2026 11:03:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:03:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:03:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:03:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:03:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:03:03.782030 21769 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/rangerkms_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/rangerkms_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.66.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.66.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:03:03 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996583, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:03:03.868619 21769 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:03:03.878417 21769 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:03:04.863597 21769 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:03:04.863677 21769 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms
I20260620 23:03:04.863689 21769 mini_ranger_kms.cc:227] Using postgres at 127.21.66.84:49851
2026-06-20 23:03:05,040 [I] DB FLAVOR :POSTGRES
2026-06-20 23:03:05,041 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:03:05,041 [I] Checking connection
2026-06-20 23:03:05,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:03:05,289 [I] connection success
2026-06-20 23:03:05,289 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:03:05,289 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:03:05,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:03:05,577 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:03:05,577 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:03:05,577 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:03:05,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:49851/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:03:05,875 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:03:06.373198 21769 mini_ranger_kms.cc:310] Created kms service
I20260620 23:03:06.595568 21769 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:03:06.640614 21769 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:03:06 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996583, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:03:07.088464 21769 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:03:07.088587 21769 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:03:07.088610 21769 mini_ranger_kms.cc:240] Using host: 127.21.66.84
I20260620 23:03:07.092046 21769 mini_ranger_kms.cc:276] Ranger KMS PID: 431
I20260620 23:03:07.092195 21769 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.66.84:53413
Jun 20, 2026 11:03:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:03:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:03:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:03:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:03:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:03:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:03:09.386283 21769 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.518s user 0.009s sys 0.150s
I20260620 23:03:09.386396 21769 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:03:09.386477 21769 mini_ranger_kms.cc:401] 127.21.66.84:53413/kms/v1/keys
Jun 20 23:03:09 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996589, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.66.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:03:10 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996589, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.66.84@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
Jun 20 23:03:10 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996589, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.66.84@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
Jun 20 23:03:12 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:03:12.817531 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:42541
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:42541
--ranger_config_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:53413/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:03:12.915937 481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:03:12.916173 481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:03:12.916199 481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:03:12.917347 481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:03:12.917397 481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:03:12.917412 481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:03:12.917428 481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:03:12.917441 481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:03:12.919052 481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.66.84:53413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:42541
--ranger_config_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:42541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:03:12.919292 481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:03:12.919842 481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:03:12.926117 486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:03:12.926154 487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:03:12.926201 481 server_base.cc:1034] running on GCE node
W20260620 23:03:12.926151 489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:03:12.926748 481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:03:12.927105 481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:03:12.928292 481 hybrid_clock.cc:648] HybridClock initialized: now 1781996592928254 us; error 55 us; skew 500 ppm
Jun 20 23:03:12 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996592, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:03:12.931166 481 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:03:12.931684 481 webserver.cc:460] Webserver started at http://127.21.66.126:43251/ using document root <none> and password file <none>
I20260620 23:03:12.931939 481 fs_manager.cc:359] Metadata directory not provided
I20260620 23:03:12.932005 481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:03:12.932171 481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:03:12 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996592, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:03:12.984603 481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "df0d6851759c4aba8c302691227412e2"
format_stamp: "Formatted at 2026-06-20 23:03:12 on dist-test-slave-qgtk"
server_key: "f0fe8643250af35fbbdd7e585185660b"
server_key_iv: "6d7425b5db1b547188a26ae9621db2c1"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:12.985011 481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "df0d6851759c4aba8c302691227412e2"
format_stamp: "Formatted at 2026-06-20 23:03:12 on dist-test-slave-qgtk"
server_key: "f0fe8643250af35fbbdd7e585185660b"
server_key_iv: "6d7425b5db1b547188a26ae9621db2c1"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:12.986434 481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 23:03:13.001482 496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:13.001775 481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:03:13.001876 481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "df0d6851759c4aba8c302691227412e2"
format_stamp: "Formatted at 2026-06-20 23:03:12 on dist-test-slave-qgtk"
server_key: "f0fe8643250af35fbbdd7e585185660b"
server_key_iv: "6d7425b5db1b547188a26ae9621db2c1"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:13.001972 481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:03:13.020358 481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:03:13.022634 481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:03:13.022796 481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:03:13.027535 481 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:42541
I20260620 23:03:13.027590 548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:42541 every 8 connection(s)
I20260620 23:03:13.027992 481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:03:13.028703 549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:03:13.031450 549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2: Bootstrap starting.
I20260620 23:03:13.032043 549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:13.032361 549 log.cc:824] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:03:13.033150 549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2: No bootstrap required, opened a new log
I20260620 23:03:13.034759 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 481
I20260620 23:03:13.034879 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:03:13.035086 21769 external_mini_cluster.cc:1428] Setting key dad4ac690f20d97591f754727baf4c21
I20260620 23:03:13.035892 549 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d6851759c4aba8c302691227412e2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42541 } }
I20260620 23:03:13.036055 549 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:03:13.036116 549 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6851759c4aba8c302691227412e2, State: Initialized, Role: FOLLOWER
I20260620 23:03:13.036257 549 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d6851759c4aba8c302691227412e2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42541 } }
I20260620 23:03:13.036420 549 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:03:13.036473 549 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:03:13.036521 549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:13.037273 549 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d6851759c4aba8c302691227412e2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42541 } }
I20260620 23:03:13.037401 549 leader_election.cc:302] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0d6851759c4aba8c302691227412e2; no voters:
I20260620 23:03:13.037569 549 leader_election.cc:288] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:03:13.037599 554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:13.037798 554 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [term 1 LEADER]: Becoming Leader. State: Replica: df0d6851759c4aba8c302691227412e2, State: Running, Role: LEADER
I20260620 23:03:13.037914 549 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:03:13.037920 554 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d6851759c4aba8c302691227412e2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42541 } }
I20260620 23:03:13.038283 554 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0d6851759c4aba8c302691227412e2. Latest consensus state: current_term: 1 leader_uuid: "df0d6851759c4aba8c302691227412e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d6851759c4aba8c302691227412e2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42541 } } }
I20260620 23:03:13.038290 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0d6851759c4aba8c302691227412e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d6851759c4aba8c302691227412e2" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 42541 } } }
I20260620 23:03:13.038357 554 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:03:13.038365 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:03:13.040030 549 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:03:13.040195 558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:03:13.045149 558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:03:13 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:03:13.049891 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:13.035659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.21.66.126:42541)
0620 23:03:13.035928 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:13.035939 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:13.035975 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:03:13.036102 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:13.036109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:13.036169 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:13.036342 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:03:13.037132 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:13.038766 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:13.040062 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:13.042810 (+ 2748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:13.042977 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:13.045963 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:03:13.045970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:03:13.045971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:03:13.045988 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:03:13.048104 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:13.048495 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:13.048505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:13.048506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:13.048596 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:13.048805 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:13.048806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:13.048806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:13.049098 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:03:13.049260 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:13.049366 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 23:03:13.049499 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260620 23:03:13.054335 558 catalog_manager.cc:1347] Generated new cluster ID: 0b226a841f984422a6584f4af5a09ba9
I20260620 23:03:13.054419 558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:03:13.088583 558 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:03:13.089174 558 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:03:13.094705 558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2: Generated new TSK 0
I20260620 23:03:13.094918 558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:03:13.817651 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:53413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42541
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:03:13.906078 594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:03:13.906359 594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:03:13.906401 594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:03:13.907693 594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:03:13.907755 594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:03:13.907801 594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:03:13.909392 594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.66.84:53413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:03:13.909680 594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:03:13.909999 594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:03:13.915315 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:03:13.915504 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:03:13.915349 602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:03:13.916069 594 server_base.cc:1034] running on GCE node
I20260620 23:03:13.916289 594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:03:13.916550 594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:03:13.919829 594 hybrid_clock.cc:648] HybridClock initialized: now 1781996593919801 us; error 41 us; skew 500 ppm
Jun 20 23:03:13 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:03:13.923115 594 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:03:13.923650 594 webserver.cc:460] Webserver started at http://127.21.66.65:41415/ using document root <none> and password file <none>
I20260620 23:03:13.923920 594 fs_manager.cc:359] Metadata directory not provided
I20260620 23:03:13.923992 594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:03:13.924181 594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:03:13 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:03:13.938848 594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "a46a652bc8f24769938feb8ab51f1725"
format_stamp: "Formatted at 2026-06-20 23:03:13 on dist-test-slave-qgtk"
server_key: "a703a6f5470782b53924b902ceb4a6d8"
server_key_iv: "c87b08afe06ef20d41ada20ed99da131"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:13.939306 594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "a46a652bc8f24769938feb8ab51f1725"
format_stamp: "Formatted at 2026-06-20 23:03:13 on dist-test-slave-qgtk"
server_key: "a703a6f5470782b53924b902ceb4a6d8"
server_key_iv: "c87b08afe06ef20d41ada20ed99da131"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:13.941582 594 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:03:13.956571 609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:13.961603 594 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 23:03:13.961786 594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "a46a652bc8f24769938feb8ab51f1725"
format_stamp: "Formatted at 2026-06-20 23:03:13 on dist-test-slave-qgtk"
server_key: "a703a6f5470782b53924b902ceb4a6d8"
server_key_iv: "c87b08afe06ef20d41ada20ed99da131"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:13.961978 594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:03:13.976900 594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:03:13.979207 594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:03:13.979393 594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:03:13.979671 594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:03:13.980170 594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:03:13.980235 594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:13.980284 594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:03:13.980314 594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:13.988463 594 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:34019
I20260620 23:03:13.988539 724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:34019 every 8 connection(s)
I20260620 23:03:13.988901 594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:03:13.994053 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 594
I20260620 23:03:13.994205 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:03:13.994436 21769 external_mini_cluster.cc:1428] Setting key 8d298cdf6d2da89f130e9328e49e8cf2
Jun 20 23:03:13 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996593, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:03:14.000283 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:13.989733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42541 (local address 127.21.66.65:50991)
0620 23:03:13.989967 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:13.990012 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:13.990580 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:13.990859 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:13.990864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:13.991071 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:03:13.991270 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:13.991274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:13.992010 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:13.992012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:13.992837 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:13.992839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:13.993301 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:13.993302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:13.993434 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:13.994223 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:03:13.994234 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:03:13.995909 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:03:13.998406 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:03:13.998415 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:03:13.998419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:03:13.998771 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:03:13.999027 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:03:13.999028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:03:13.999028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:03:13.999167 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:03:13.999485 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:03:13.999491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:03:13.999681 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 23:03:13.999789 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:03:13.999971 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260620 23:03:14.001210 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:13.989735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:50991 (local address 127.21.66.126:42541)
0620 23:03:13.989894 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:13.989896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:13.990157 (+ 261us) server_negotiation.cc:407] Connection header received
0620 23:03:13.990703 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:13.990704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:13.990720 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:13.990784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:03:13.991369 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:13.991901 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:13.992948 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:13.993182 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:13.993290 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:13.996033 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:03:13.996035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:03:13.996035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:03:13.996044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:03:13.998276 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:13.998873 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:13.998874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:13.998874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:13.998942 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:13.999246 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:13.999246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:13.999247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:13.999395 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:03:14.000936 (+ 1541us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.000986 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.001105 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260620 23:03:14.001729 725 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42541
I20260620 23:03:14.001843 725 heartbeater.cc:464] Registering TS with master...
I20260620 23:03:14.002034 725 heartbeater.cc:511] Master 127.21.66.126:42541 requested a full tablet report, sending...
I20260620 23:03:14.002629 513 ts_manager.cc:194] Registered new tserver with Master: a46a652bc8f24769938feb8ab51f1725 (127.21.66.65:34019)
I20260620 23:03:14.003257 513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:50991
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:03:14.029827 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:53413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42541
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:03:14.146649 731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:03:14.146919 731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:03:14.146997 731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:03:14.148726 731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:03:14.148778 731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:03:14.148818 731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:03:14.150980 731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.66.84:53413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:03:14.151245 731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:03:14.151546 731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:03:14.158139 738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:03:14.158324 739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:03:14.158533 741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:03:14.159962 731 server_base.cc:1034] running on GCE node
I20260620 23:03:14.160209 731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:03:14.160506 731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:03:14.161701 731 hybrid_clock.cc:648] HybridClock initialized: now 1781996594161667 us; error 49 us; skew 500 ppm
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:03:14.165171 731 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:03:14.165823 731 webserver.cc:460] Webserver started at http://127.21.66.66:36563/ using document root <none> and password file <none>
I20260620 23:03:14.166188 731 fs_manager.cc:359] Metadata directory not provided
I20260620 23:03:14.166263 731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:03:14.166415 731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:03:14.182509 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "44c6423d173f4bf0a10017af31cf7436"
format_stamp: "Formatted at 2026-06-20 23:03:14 on dist-test-slave-qgtk"
server_key: "43eba4e96fa9fab6177db08b64071dcf"
server_key_iv: "15e9475b2c5910b7eb299ef5e6fbde2f"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:14.183030 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "44c6423d173f4bf0a10017af31cf7436"
format_stamp: "Formatted at 2026-06-20 23:03:14 on dist-test-slave-qgtk"
server_key: "43eba4e96fa9fab6177db08b64071dcf"
server_key_iv: "15e9475b2c5910b7eb299ef5e6fbde2f"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:14.186553 731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:03:14.205914 748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:14.206260 731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:03:14.206367 731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "44c6423d173f4bf0a10017af31cf7436"
format_stamp: "Formatted at 2026-06-20 23:03:14 on dist-test-slave-qgtk"
server_key: "43eba4e96fa9fab6177db08b64071dcf"
server_key_iv: "15e9475b2c5910b7eb299ef5e6fbde2f"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:14.206501 731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:03:14.227240 731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:03:14.233582 731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:03:14.233825 731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:03:14.234177 731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:03:14.234642 731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:03:14.234737 731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:14.234813 731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:03:14.234875 731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:14.243248 731 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:37971
I20260620 23:03:14.243321 861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:37971 every 8 connection(s)
I20260620 23:03:14.243855 731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:03:14.246443 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 731
I20260620 23:03:14.246567 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:03:14.246765 21769 external_mini_cluster.cc:1428] Setting key 69c18ec34583d09c3d579aa14e2d37e5
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:03:14.256268 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.245086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:50089 (local address 127.21.66.126:42541)
0620 23:03:14.245133 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.245136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.245147 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:03:14.245881 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.245883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.245899 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.245975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:03:14.246882 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.247737 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.249518 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.249853 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.249904 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.252902 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:03:14.252903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:03:14.252904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:03:14.252912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:03:14.254690 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:14.255170 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:14.255171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:14.255172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:14.255237 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:14.255514 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:14.255515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:14.255515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:14.255702 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:03:14.255773 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.256075 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.256168 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:03:14.256489 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.244775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42541 (local address 127.21.66.66:50089)
0620 23:03:14.244991 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.245048 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:14.245732 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:14.246078 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:14.246083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:14.246429 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:03:14.246743 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.246751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.247986 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.247987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.249404 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.249407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.250009 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.250011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.250225 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.250961 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:03:14.250972 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:03:14.252784 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:03:14.254801 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:03:14.254808 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:03:14.254811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:03:14.255070 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:03:14.255314 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:03:14.255315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:03:14.255316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:03:14.255439 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:03:14.255764 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:03:14.255770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:03:14.255926 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 23:03:14.256054 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:03:14.256212 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260620 23:03:14.257186 862 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42541
I20260620 23:03:14.257306 862 heartbeater.cc:464] Registering TS with master...
I20260620 23:03:14.257498 862 heartbeater.cc:511] Master 127.21.66.126:42541 requested a full tablet report, sending...
I20260620 23:03:14.258000 513 ts_manager.cc:194] Registered new tserver with Master: 44c6423d173f4bf0a10017af31cf7436 (127.21.66.66:37971)
I20260620 23:03:14.258458 513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:50089
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:03:14.289628 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:53413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42541
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:36887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:03:14.403456 868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:03:14.403851 868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:03:14.403946 868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:03:14.405854 868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:03:14.405994 868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:03:14.406082 868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:03:14.408550 868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:36887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.66.84:53413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:42541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:03:14.409070 868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:03:14.409415 868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:03:14.416380 874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:03:14.416785 876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:03:14.416996 873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:03:14.418303 868 server_base.cc:1034] running on GCE node
I20260620 23:03:14.418550 868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:03:14.418833 868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:03:14.420007 868 hybrid_clock.cc:648] HybridClock initialized: now 1781996594419990 us; error 28 us; skew 500 ppm
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:03:14.423460 868 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:03:14.423990 868 webserver.cc:460] Webserver started at http://127.21.66.67:34897/ using document root <none> and password file <none>
I20260620 23:03:14.424268 868 fs_manager.cc:359] Metadata directory not provided
I20260620 23:03:14.424396 868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:03:14.424582 868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:03:14.440313 868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "fc8d00e177ba4234b9385c3e3277d741"
format_stamp: "Formatted at 2026-06-20 23:03:14 on dist-test-slave-qgtk"
server_key: "5099b17ba95755ddc9c8b18be50efcf6"
server_key_iv: "1a6f574a2f378d420b272db846fcbb7c"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:14.442502 868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "fc8d00e177ba4234b9385c3e3277d741"
format_stamp: "Formatted at 2026-06-20 23:03:14 on dist-test-slave-qgtk"
server_key: "5099b17ba95755ddc9c8b18be50efcf6"
server_key_iv: "1a6f574a2f378d420b272db846fcbb7c"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:14.445187 868 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:03:14.462073 883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:14.462481 868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:03:14.462608 868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "fc8d00e177ba4234b9385c3e3277d741"
format_stamp: "Formatted at 2026-06-20 23:03:14 on dist-test-slave-qgtk"
server_key: "5099b17ba95755ddc9c8b18be50efcf6"
server_key_iv: "1a6f574a2f378d420b272db846fcbb7c"
server_key_version: "kuduclusterkey@0"
I20260620 23:03:14.462723 868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:03:14.497557 868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:03:14.499943 868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:03:14.500146 868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:03:14.500438 868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:03:14.500903 868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:03:14.500962 868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:14.501031 868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:03:14.501061 868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:03:14.508387 868 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:46733
I20260620 23:03:14.508929 868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:03:14.509485 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:46733 every 8 connection(s)
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:03:14.517771 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 868
I20260620 23:03:14.517910 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:03:14.518178 21769 external_mini_cluster.cc:1428] Setting key 7ab39b51837d7ff7e3e29ba1cf24d6dc
I20260620 23:03:14.520785 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.510250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:46703 (local address 127.21.66.126:42541)
0620 23:03:14.510296 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.510298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.510575 (+ 277us) server_negotiation.cc:407] Connection header received
0620 23:03:14.511240 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.511241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.511255 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.511297 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:03:14.511888 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.512547 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.513613 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.513833 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.513925 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.516633 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:03:14.516635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:03:14.516635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:03:14.516644 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:03:14.518553 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:14.519548 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:14.519549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:14.519550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:14.519620 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:14.519988 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:14.519988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:14.519989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:14.520119 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:03:14.520197 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.520589 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.520687 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:03:14.521643 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.510270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:42541 (local address 127.21.66.67:46703)
0620 23:03:14.510448 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.510483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:14.511126 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:14.511371 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:14.511375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:14.511585 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:03:14.511790 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.511794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.512684 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.512685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.513515 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.513516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.513938 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.513939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.514075 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.514725 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:03:14.514734 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:03:14.516498 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:03:14.518861 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:03:14.518871 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:03:14.518875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:03:14.519453 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:03:14.519694 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:03:14.519694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:03:14.519695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:03:14.519869 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:03:14.520189 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:03:14.520195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:03:14.520415 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 23:03:14.521193 (+ 778us) client_negotiation.cc:240] Negotiation successful
0620 23:03:14.521395 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260620 23:03:14.522330 997 heartbeater.cc:347] Connected to a master server at 127.21.66.126:42541
I20260620 23:03:14.522464 997 heartbeater.cc:464] Registering TS with master...
I20260620 23:03:14.522656 997 heartbeater.cc:511] Master 127.21.66.126:42541 requested a full tablet report, sending...
I20260620 23:03:14.523063 513 ts_manager.cc:194] Registered new tserver with Master: fc8d00e177ba4234b9385c3e3277d741 (127.21.66.67:46733)
I20260620 23:03:14.523567 513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:46703
I20260620 23:03:14.530284 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:03:14.541416 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.532026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.21.66.126:42541)
0620 23:03:14.532076 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.532078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.532960 (+ 882us) server_negotiation.cc:407] Connection header received
0620 23:03:14.534274 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.534275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.534287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.534357 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:03:14.534965 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.535526 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.536408 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.536601 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.536704 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.538305 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:03:14.538307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:03:14.538307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:03:14.538315 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:03:14.539996 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:14.540460 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:14.540462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:14.540462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:14.540521 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:03:14.540737 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:03:14.540738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:03:14.540744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:03:14.540888 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:03:14.540991 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.541214 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.541303 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:03:14.543203 513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:03:14.545017 513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:03:14.555656 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.548202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.21.66.67:46733)
0620 23:03:14.548379 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.548385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.550314 (+ 1929us) server_negotiation.cc:407] Connection header received
0620 23:03:14.550881 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.550884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.550901 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.550959 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:03:14.551183 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.552272 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.554315 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.555113 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.555167 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.555251 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.555507 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.555547 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260620 23:03:14.556200 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.547676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:46733 (local address 127.0.0.1:37860)
0620 23:03:14.550245 (+ 2569us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.550256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:14.550465 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:14.551065 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:14.551065 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:14.551067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:03:14.551103 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.551104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.552375 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.552376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.554189 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.554192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.555309 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.555310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.555387 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.555401 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:03:14.556053 (+ 652us) client_negotiation.cc:240] Negotiation successful
0620 23:03:14.556121 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2549,"mutex_wait_us":499,"thread_start_us":89,"threads_started":1}
I20260620 23:03:14.557123 929 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9efb47eaa64d61a60f6b1ceb70b326 (DEFAULT_TABLE table=test-table [id=f6321ed644ee403690440a724557f910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:03:14.557519 929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9efb47eaa64d61a60f6b1ceb70b326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:03:14.560664 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.547676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34019 (local address 127.0.0.1:36150)
0620 23:03:14.548660 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.550103 (+ 1443us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:14.550265 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:14.553520 (+ 3255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:14.553521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:14.553523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:03:14.553565 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.553567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.558019 (+ 4452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.558020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.559119 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.559122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.560424 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.560425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.560487 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.560490 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:03:14.560534 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 23:03:14.560581 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":114,"threads_started":1}
I20260620 23:03:14.561002 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.547951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.21.66.65:34019)
0620 23:03:14.552667 (+ 4716us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.552673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.552683 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:03:14.552723 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.552725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.552740 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.556766 (+ 4026us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:03:14.556840 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.557921 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.559517 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.560251 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.560325 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.560387 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.560873 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.560928 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4680,"thread_start_us":55,"threads_started":1}
I20260620 23:03:14.561761 656 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9efb47eaa64d61a60f6b1ceb70b326 (DEFAULT_TABLE table=test-table [id=f6321ed644ee403690440a724557f910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:03:14.562222 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9efb47eaa64d61a60f6b1ceb70b326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:03:14.562722 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.548183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34788 (local address 127.21.66.66:37971)
0620 23:03:14.548454 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.548458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.550422 (+ 1964us) server_negotiation.cc:407] Connection header received
0620 23:03:14.550491 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.550494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.550511 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.550584 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:03:14.551132 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.552301 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.555278 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.556349 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.556413 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.556488 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.562613 (+ 6125us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.562660 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":79,"threads_started":1}
I20260620 23:03:14.562943 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.547903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:37971 (local address 127.0.0.1:34788)
0620 23:03:14.550362 (+ 2459us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.550370 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:14.550417 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:14.550956 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:14.550959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:14.550962 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:03:14.551046 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.551049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.552387 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.552387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.555155 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.555158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.556523 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.556524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.556592 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.556595 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:03:14.556641 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 23:03:14.562869 (+ 6228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2446,"mutex_wait_us":50,"thread_start_us":64,"threads_started":1}
I20260620 23:03:14.563220 1015 tablet_bootstrap.cc:492] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Bootstrap starting.
I20260620 23:03:14.563640 794 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9efb47eaa64d61a60f6b1ceb70b326 (DEFAULT_TABLE table=test-table [id=f6321ed644ee403690440a724557f910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:03:14.564036 794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9efb47eaa64d61a60f6b1ceb70b326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:03:14.564392 1015 tablet_bootstrap.cc:654] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:14.564742 1015 log.cc:824] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Log is configured to *not* fsync() on all Append() calls
I20260620 23:03:14.565917 1017 tablet_bootstrap.cc:492] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725: Bootstrap starting.
I20260620 23:03:14.566653 1017 tablet_bootstrap.cc:654] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:14.567003 1017 log.cc:824] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725: Log is configured to *not* fsync() on all Append() calls
I20260620 23:03:14.567940 1018 tablet_bootstrap.cc:492] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Bootstrap starting.
I20260620 23:03:14.568938 1018 tablet_bootstrap.cc:654] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Neither blocks nor log segments found. Creating new log.
I20260620 23:03:14.569249 1018 log.cc:824] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Log is configured to *not* fsync() on all Append() calls
I20260620 23:03:14.570147 1017 tablet_bootstrap.cc:492] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725: No bootstrap required, opened a new log
I20260620 23:03:14.570264 1017 ts_tablet_manager.cc:1397] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:03:14.570302 1018 tablet_bootstrap.cc:492] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: No bootstrap required, opened a new log
I20260620 23:03:14.570432 1018 ts_tablet_manager.cc:1397] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:03:14.572968 1017 raft_consensus.cc:348] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.573185 1017 raft_consensus.cc:374] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:03:14.573240 1017 raft_consensus.cc:729] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46a652bc8f24769938feb8ab51f1725, State: Initialized, Role: FOLLOWER
I20260620 23:03:14.573100 1018 raft_consensus.cc:348] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.573396 1018 raft_consensus.cc:374] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:03:14.573441 1018 raft_consensus.cc:729] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c6423d173f4bf0a10017af31cf7436, State: Initialized, Role: FOLLOWER
I20260620 23:03:14.573545 1017 consensus_queue.cc:260] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.573573 1018 consensus_queue.cc:260] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.573909 1017 ts_tablet_manager.cc:1428] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 23:03:14.573943 1018 ts_tablet_manager.cc:1428] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:03:14.574105 725 heartbeater.cc:503] Master 127.21.66.126:42541 was elected leader, sending a full tablet report...
I20260620 23:03:14.574182 862 heartbeater.cc:503] Master 127.21.66.126:42541 was elected leader, sending a full tablet report...
I20260620 23:03:14.576125 1015 tablet_bootstrap.cc:492] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: No bootstrap required, opened a new log
I20260620 23:03:14.576223 1015 ts_tablet_manager.cc:1397] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260620 23:03:14.577909 1015 raft_consensus.cc:348] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.578066 1015 raft_consensus.cc:374] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:03:14.578111 1015 raft_consensus.cc:729] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8d00e177ba4234b9385c3e3277d741, State: Initialized, Role: FOLLOWER
I20260620 23:03:14.578272 1015 consensus_queue.cc:260] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.578560 1015 ts_tablet_manager.cc:1428] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:03:14.578647 997 heartbeater.cc:503] Master 127.21.66.126:42541 was elected leader, sending a full tablet report...
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
W20260620 23:03:14.740018 726 tablet.cc:2369] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:03:14.745803 863 tablet.cc:2369] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:03:14.760330 998 tablet.cc:2369] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:03:14.848170 1023 raft_consensus.cc:482] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:03:14.848326 1023 raft_consensus.cc:504] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.848732 1023 leader_election.cc:288] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46a652bc8f24769938feb8ab51f1725 (127.21.66.65:34019), 44c6423d173f4bf0a10017af31cf7436 (127.21.66.66:37971)
I20260620 23:03:14.852540 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.848958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:37971 (local address 127.21.66.67:52473)
0620 23:03:14.849054 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.849065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:14.849107 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:14.849397 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:14.849398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:14.849400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:03:14.849465 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.849467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.850414 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.850415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.851586 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.851589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.852316 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.852317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.852384 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.852387 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:03:14.852430 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 23:03:14.852468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 23:03:14.852819 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.849180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:52473 (local address 127.21.66.66:37971)
0620 23:03:14.849223 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.849225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.849232 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:03:14.849258 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.849259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.849268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.849311 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:03:14.849532 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.850320 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.851714 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.852206 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.852275 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.852286 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.852724 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.852762 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:03:14.853065 798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9efb47eaa64d61a60f6b1ceb70b326" candidate_uuid: "fc8d00e177ba4234b9385c3e3277d741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c6423d173f4bf0a10017af31cf7436" is_pre_election: true
I20260620 23:03:14.853209 798 raft_consensus.cc:2454] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8d00e177ba4234b9385c3e3277d741 in term 0.
I20260620 23:03:14.853439 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.849196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:41325 (local address 127.21.66.65:34019)
0620 23:03:14.849368 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.849370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.849416 (+ 46us) server_negotiation.cc:407] Connection header received
0620 23:03:14.849533 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.849534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.849547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.849588 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:03:14.849842 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.850592 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.851947 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.852542 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.852596 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.852616 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.853317 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.853366 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260620 23:03:14.853425 887 leader_election.cc:302] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c6423d173f4bf0a10017af31cf7436, fc8d00e177ba4234b9385c3e3277d741; no voters:
I20260620 23:03:14.853560 1023 raft_consensus.cc:2781] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:03:14.853600 1023 raft_consensus.cc:482] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:03:14.853621 1023 raft_consensus.cc:3037] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:14.853956 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.848982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34019 (local address 127.21.66.67:41325)
0620 23:03:14.849340 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.849353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:03:14.849446 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:03:14.849676 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:03:14.849677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:03:14.849679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:03:14.849760 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.849762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.850719 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.850720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.851825 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:03:14.851828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.853029 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.853030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:03:14.853103 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.853107 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:03:14.853795 (+ 688us) client_negotiation.cc:240] Negotiation successful
0620 23:03:14.853846 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":95,"threads_started":1}
I20260620 23:03:14.854440 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9efb47eaa64d61a60f6b1ceb70b326" candidate_uuid: "fc8d00e177ba4234b9385c3e3277d741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46a652bc8f24769938feb8ab51f1725" is_pre_election: true
I20260620 23:03:14.854543 1023 raft_consensus.cc:504] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.854617 661 raft_consensus.cc:2454] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8d00e177ba4234b9385c3e3277d741 in term 0.
I20260620 23:03:14.854722 1023 leader_election.cc:288] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [CANDIDATE]: Term 1 election: Requested vote from peers a46a652bc8f24769938feb8ab51f1725 (127.21.66.65:34019), 44c6423d173f4bf0a10017af31cf7436 (127.21.66.66:37971)
I20260620 23:03:14.854837 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9efb47eaa64d61a60f6b1ceb70b326" candidate_uuid: "fc8d00e177ba4234b9385c3e3277d741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46a652bc8f24769938feb8ab51f1725"
I20260620 23:03:14.854915 661 raft_consensus.cc:3037] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:14.854928 798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9efb47eaa64d61a60f6b1ceb70b326" candidate_uuid: "fc8d00e177ba4234b9385c3e3277d741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c6423d173f4bf0a10017af31cf7436"
I20260620 23:03:14.855013 798 raft_consensus.cc:3037] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:03:14.855718 661 raft_consensus.cc:2454] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8d00e177ba4234b9385c3e3277d741 in term 1.
I20260620 23:03:14.855912 885 leader_election.cc:302] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a46a652bc8f24769938feb8ab51f1725, fc8d00e177ba4234b9385c3e3277d741; no voters:
I20260620 23:03:14.856003 798 raft_consensus.cc:2454] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8d00e177ba4234b9385c3e3277d741 in term 1.
I20260620 23:03:14.856178 1023 raft_consensus.cc:2781] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:03:14.856331 1023 raft_consensus.cc:686] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 1 LEADER]: Becoming Leader. State: Replica: fc8d00e177ba4234b9385c3e3277d741, State: Running, Role: LEADER
I20260620 23:03:14.856447 1023 consensus_queue.cc:237] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } }
I20260620 23:03:14.857323 512 catalog_manager.cc:5495] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc8d00e177ba4234b9385c3e3277d741 (127.21.66.67). New cstate: current_term: 1 leader_uuid: "fc8d00e177ba4234b9385c3e3277d741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8d00e177ba4234b9385c3e3277d741" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 } health_report { overall_health: UNKNOWN } } }
I20260620 23:03:14.910583 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.907015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.21.66.67:46733)
0620 23:03:14.907079 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 23:03:14.907081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:03:14.907091 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:03:14.907137 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:03:14.907138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:03:14.907162 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:03:14.907251 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:03:14.907484 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.908205 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.909366 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:03:14.909681 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:03:14.909754 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:03:14.909989 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:03:14.910159 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:03:14.910247 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:03:14.910471 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 23:03:14.910495 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:03:14.912748 661 raft_consensus.cc:1261] T 2e9efb47eaa64d61a60f6b1ceb70b326 P a46a652bc8f24769938feb8ab51f1725 [term 1 FOLLOWER]: Refusing update from remote peer fc8d00e177ba4234b9385c3e3277d741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:03:14.913352 798 raft_consensus.cc:1261] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 1 FOLLOWER]: Refusing update from remote peer fc8d00e177ba4234b9385c3e3277d741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:03:14.913429 1025 consensus_queue.cc:1035] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46a652bc8f24769938feb8ab51f1725" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:03:14.913767 1023 consensus_queue.cc:1035] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c6423d173f4bf0a10017af31cf7436" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:03:14.916453 1028 mvcc.cc:205] Tried to move back new op lower bound from 7299058052759298048 to 7299058052532236288. Current Snapshot: MvccSnapshot[applied={T|T < 7299058052759298048}]
I20260620 23:03:14.918241 1031 mvcc.cc:205] Tried to move back new op lower bound from 7299058052759298048 to 7299058052532236288. Current Snapshot: MvccSnapshot[applied={T|T < 7299058052759298048}]
I20260620 23:03:14.919519 1030 mvcc.cc:205] Tried to move back new op lower bound from 7299058052759298048 to 7299058052532236288. Current Snapshot: MvccSnapshot[applied={T|T < 7299058052759298048 or (T in {7299058052759298048})}]
I20260620 23:03:14.919703 511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:03:14.919795 511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:03:14.920579 511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df0d6851759c4aba8c302691227412e2: Sending DeleteTablet for 3 replicas of tablet 2e9efb47eaa64d61a60f6b1ceb70b326
I20260620 23:03:14.920962 656 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9efb47eaa64d61a60f6b1ceb70b326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:03:14 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:36150
I20260620 23:03:14.921061 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9efb47eaa64d61a60f6b1ceb70b326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:03:14 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:34788
I20260620 23:03:14.921214 929 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9efb47eaa64d61a60f6b1ceb70b326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:03:14 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:37860
I20260620 23:03:14.921483 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 594
I20260620 23:03:14.926281 1039 tablet_replica.cc:330] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: stopping tablet replica
I20260620 23:03:14.926419 1039 raft_consensus.cc:2229] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:03:14.926546 1039 raft_consensus.cc:2258] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:14.927081 1039 ts_tablet_manager.cc:1905] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:03:14.931082 1039 ts_tablet_manager.cc:1918] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:03:14.931185 1039 log.cc:1196] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/2e9efb47eaa64d61a60f6b1ceb70b326
I20260620 23:03:14.931466 1039 ts_tablet_manager.cc:1939] T 2e9efb47eaa64d61a60f6b1ceb70b326 P 44c6423d173f4bf0a10017af31cf7436: Deleting consensus metadata
I20260620 23:03:14.931726 1040 tablet_replica.cc:330] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: stopping tablet replica
I20260620 23:03:14.931859 1040 raft_consensus.cc:2229] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:03:14.931982 1040 raft_consensus.cc:2258] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:03:14.934552 500 catalog_manager.cc:4841] TS 44c6423d173f4bf0a10017af31cf7436 (127.21.66.66:37971): tablet 2e9efb47eaa64d61a60f6b1ceb70b326 (table test-table [id=f6321ed644ee403690440a724557f910]) successfully deleted
I20260620 23:03:14.934815 1040 ts_tablet_manager.cc:1905] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:03:14.936442 1040 ts_tablet_manager.cc:1918] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:03:14.936522 1040 log.cc:1196] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/2e9efb47eaa64d61a60f6b1ceb70b326
W20260620 23:03:14.937134 498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:34019 (error 108)
I20260620 23:03:14.936803 1040 ts_tablet_manager.cc:1939] T 2e9efb47eaa64d61a60f6b1ceb70b326 P fc8d00e177ba4234b9385c3e3277d741: Deleting consensus metadata
I20260620 23:03:14.938498 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 731
I20260620 23:03:14.946718 500 catalog_manager.cc:4841] TS fc8d00e177ba4234b9385c3e3277d741 (127.21.66.67:46733): tablet 2e9efb47eaa64d61a60f6b1ceb70b326 (table test-table [id=f6321ed644ee403690440a724557f910]) successfully deleted
I20260620 23:03:14.947906 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:03:14.947775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:34019 (local address 127.0.0.1:36156)
0620 23:03:14.947820 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:03:14.947861 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:34019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 23:03:14.947928 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 868
W20260620 23:03:14.949527 498 catalog_manager.cc:4584] TS a46a652bc8f24769938feb8ab51f1725 (127.21.66.65:34019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e9efb47eaa64d61a60f6b1ceb70b326: Network error: Client connection negotiation failed: client connection to 127.21.66.65:34019: connect: Connection refused (error 111)
Jun 20 23:03:14 dist-test-slave-qgtk krb5kdc[32505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996594, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:03:14.954282 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 481
I20260620 23:03:14.959764 21769 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:03:15.298761 21769 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:03:15.298913 21769 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:03:15.644209 21769 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:03:15.644 UTC [32535] LOG: received smart shutdown request
2026-06-20 23:03:15.648 UTC [32535] LOG: background worker "logical replication launcher" (PID 32544) exited with exit code 1
2026-06-20 23:03:15.648 UTC [32539] LOG: shutting down
2026-06-20 23:03:15.840 UTC [32535] LOG: database system is shut down
2026-06-20T23:03:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50565 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:03:16 dist-test-slave-qgtk krb5kdc[1050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:03:16 dist-test-slave-qgtk krb5kdc[1050](info): set up 2 sockets
Jun 20 23:03:16 dist-test-slave-qgtk krb5kdc[1050](info): commencing operation
krb5kdc: starting...
W20260620 23:03:17.042063 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:03:17 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:03:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/rangeradmin_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/rangeradmin_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/rangerlookup_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/rangerlookup_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
I20260620 23:03:17.225522 21769 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxQfmfq/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/postgres -l logfile start
2026-06-20 23:03:19.869 UTC [1080] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:03:19.869 UTC [1080] LOG: listening on IPv4 address "127.21.66.84", port 37211
2026-06-20 23:03:19.873 UTC [1080] LOG: listening on Unix socket "/tmp/.s.PGSQL.37211"
2026-06-20 23:03:19.911 UTC [1083] LOG: database system was shut down at 2026-06-20 23:03:18 UTC
2026-06-20 23:03:19.916 UTC [1080] LOG: database system is ready to accept connections
I20260620 23:03:20.870059 21769 mini_postgres.cc:96] Postgres bound to 37211
2026-06-20 23:03:20.876 UTC [1091] FATAL: database "slave" does not exist
127.21.66.84:37211 - accepting connections
I20260620 23:03:20.876900 21769 mini_ranger.cc:161] Starting Ranger...
I20260620 23:03:20.893268 21769 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:03:21.419373 21769 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:03:21.419476 21769 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-admin
2026-06-20 23:03:21,693 [I] DB FLAVOR :POSTGRES
2026-06-20 23:03:21,694 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:03:21,694 [I] Checking connection..
2026-06-20 23:03:21,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:03:21,961 [I] Checking connection passed.
2026-06-20 23:03:21,961 [I] --------- Verifying version history table ---------
2026-06-20 23:03:21,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:03:22,241 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:03:22,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:03:22,510 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:03:22,510 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:03:22,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:03:22.778 UTC [1155] WARNING: there is no transaction in progress
2026-06-20 23:03:22,785 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:03:22,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:03:23,045 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:03:23,045 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:03:23,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:03:23,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:03:23,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qgtk.c.gcp-upstream.internal','N') ;"
2026-06-20 23:03:23,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:03:24,071 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:03:24,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:03:24,315 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:03:24,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:03:24,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:03:24,852 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:03:24,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:03:25.922 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:25.939 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:25.958 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:25.976 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:25.996 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.017 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.080 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.088 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.105 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.118 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.129 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.145 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.155 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.162 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.170 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26.178 UTC [1263] WARNING: there is no transaction in progress
2026-06-20 23:03:26,899 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:03:26,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:03:27,146 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:03:27,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:03:27,392 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:03:27,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:03:27,631 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:03:27,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:03:27,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:03:28,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qgtk.c.gcp-upstream.internal';"
2026-06-20 23:03:28,398 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:03:28,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:03:28,633 [I] DB_PATCHES have already been applied
I20260620 23:03:28.638751 21769 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxQfmfq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:03:28.638820 21769 mini_ranger.cc:193] Using host: 127.21.66.84
I20260620 23:03:28.641911 21769 mini_ranger.cc:227] Ranger admin URL: http://127.21.66.84:43227
Jun 20, 2026 11:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:03:29.852875 21769 mini_ranger.cc:160] Time spent starting Ranger: real 8.976s user 0.000s sys 0.010s
Jun 20, 2026 11:03:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:03:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:03:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:03:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:03:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:03:53.454476 21769 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.66.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.66.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/rangerkms_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.66.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/rangerkms_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.66.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.66.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.66.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/HTTP_127.21.66.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:03:53 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996633, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:03:53.542474 21769 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:03:53.553134 21769 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:03:54.563175 21769 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:03:54.563287 21769 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms
I20260620 23:03:54.563318 21769 mini_ranger_kms.cc:227] Using postgres at 127.21.66.84:37211
2026-06-20 23:03:54,727 [I] DB FLAVOR :POSTGRES
2026-06-20 23:03:54,727 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:03:54,727 [I] Checking connection
2026-06-20 23:03:54,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:03:54,999 [I] connection success
2026-06-20 23:03:54,999 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:03:54,999 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:03:54,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:03:55,240 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:03:55,240 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:03:55,240 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:03:55,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.66.84:37211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:03:55,532 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:03:56.095212 21769 mini_ranger_kms.cc:310] Created kms service
I20260620 23:03:56.286865 21769 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:03:56.336700 21769 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:03:56 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996633, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:03:56.790850 21769 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:03:56.791069 21769 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxQfmfq/build/release/bin/postgresql.jar:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxQfmfq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:03:56.791091 21769 mini_ranger_kms.cc:240] Using host: 127.21.66.84
I20260620 23:03:56.795029 21769 mini_ranger_kms.cc:276] Ranger KMS PID: 1444
I20260620 23:03:56.795118 21769 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.66.84:34459
Jun 20, 2026 11:03:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:03:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:03:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:03:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:03:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:03:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:03:59.110457 21769 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.568s user 0.012s sys 0.135s
I20260620 23:03:59.110548 21769 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:03:59.110584 21769 mini_ranger_kms.cc:401] 127.21.66.84:34459/kms/v1/keys
Jun 20 23:03:59 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996639, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.66.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:03:59 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996639, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.66.84@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
Jun 20 23:03:59 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996639, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.66.84@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
Jun 20 23:04:02 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:04:02.219092 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45919
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:45919
--ranger_config_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:02.303731 1494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:02.303958 1494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:02.303982 1494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:02.305306 1494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:02.305362 1494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:02.305377 1494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:02.305389 1494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:02.305401 1494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:02.307188 1494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.66.84:34459/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:45919
--ranger_config_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:02.307474 1494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:02.307749 1494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:02.313406 1499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:02.313442 1494 server_base.cc:1034] running on GCE node
W20260620 23:04:02.313524 1502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:02.313442 1500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:02.313997 1494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:02.314378 1494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:02.315557 1494 hybrid_clock.cc:648] HybridClock initialized: now 1781996642315531 us; error 40 us; skew 500 ppm
Jun 20 23:04:02 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996642, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:02.323078 1494 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:02.323683 1494 webserver.cc:460] Webserver started at http://127.21.66.126:36181/ using document root <none> and password file <none>
I20260620 23:04:02.323989 1494 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:02.324054 1494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:02.324259 1494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:04:02 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996642, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:02.387890 1494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "8e9152ca98d5468e9659968477648533"
format_stamp: "Formatted at 2026-06-20 23:04:02 on dist-test-slave-qgtk"
server_key: "62625a5c4242a49bc2040db046664d40"
server_key_iv: "053fb01bd9cab13cfd441ea3278f2fa9"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:02.388315 1494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "8e9152ca98d5468e9659968477648533"
format_stamp: "Formatted at 2026-06-20 23:04:02 on dist-test-slave-qgtk"
server_key: "62625a5c4242a49bc2040db046664d40"
server_key_iv: "053fb01bd9cab13cfd441ea3278f2fa9"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:02.389801 1494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 23:04:02.404695 1509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:02.404975 1494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:02.405102 1494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "8e9152ca98d5468e9659968477648533"
format_stamp: "Formatted at 2026-06-20 23:04:02 on dist-test-slave-qgtk"
server_key: "62625a5c4242a49bc2040db046664d40"
server_key_iv: "053fb01bd9cab13cfd441ea3278f2fa9"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:02.405213 1494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:02.420943 1494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:02.423460 1494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:02.423743 1494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:02.429400 1494 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:45919
I20260620 23:04:02.429489 1561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:45919 every 8 connection(s)
I20260620 23:04:02.429908 1494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:02.430759 1562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:02.434165 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Bootstrap starting.
I20260620 23:04:02.434921 1562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:02.435279 1562 log.cc:824] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:02.435753 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 1494
I20260620 23:04:02.435873 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:04:02.436026 21769 external_mini_cluster.cc:1428] Setting key 4848707668688eb1e82e279a6c4c676a
I20260620 23:04:02.436235 1562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: No bootstrap required, opened a new log
I20260620 23:04:02.439342 1562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:02.439684 1562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:02.439767 1562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9152ca98d5468e9659968477648533, State: Initialized, Role: FOLLOWER
I20260620 23:04:02.439970 1562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:02.440086 1562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:02.440122 1562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:02.440167 1562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:02.441001 1562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:02.441130 1562 leader_election.cc:302] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e9152ca98d5468e9659968477648533; no voters:
I20260620 23:04:02.441352 1562 leader_election.cc:288] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:04:02.441406 1567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:02.441722 1562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:02.442797 1567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 LEADER]: Becoming Leader. State: Replica: 8e9152ca98d5468e9659968477648533, State: Running, Role: LEADER
I20260620 23:04:02.442966 1567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:02.444048 1567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e9152ca98d5468e9659968477648533. Latest consensus state: current_term: 1 leader_uuid: "8e9152ca98d5468e9659968477648533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } } }
I20260620 23:04:02.444213 1567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:02.444640 1562 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:04:02.444687 1582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:02.445060 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e9152ca98d5468e9659968477648533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } } }
I20260620 23:04:02.445129 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:02.445247 1582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:04:02 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:02.452698 1582 catalog_manager.cc:1347] Generated new cluster ID: 5dbc35f6990d47919958b6a9069a08ed
I20260620 23:04:02.452824 1582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:02.453694 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:02.436571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.21.66.126:45919)
0620 23:04:02.436830 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:02.436839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:02.436861 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:04:02.439318 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:02.439329 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:02.439392 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:02.439572 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:02.440402 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:02.441564 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:02.442714 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:02.443355 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:02.443494 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:02.447346 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:02.447403 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:02.447403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:02.447537 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:02.451154 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:02.451701 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:02.451705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:02.451706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:02.451840 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:02.452121 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:02.452126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:02.452127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:02.452583 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:02.452761 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:02.453003 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 23:04:02.453434 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260620 23:04:02.479415 1582 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:04:02.480279 1582 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:02.483289 1582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Generated new TSK 0
I20260620 23:04:02.483531 1582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:04:03.185310 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45919
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:03.302032 1608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:03.302387 1608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:03.302417 1608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:03.303778 1608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:03.303825 1608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:03.303855 1608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:04:03.305653 1608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.66.84:34459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:03.305904 1608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:03.306236 1608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:03.312482 1616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:03.312539 1614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:03.312691 1613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:03.313680 1608 server_base.cc:1034] running on GCE node
I20260620 23:04:03.313915 1608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:03.314232 1608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:03.315447 1608 hybrid_clock.cc:648] HybridClock initialized: now 1781996643315426 us; error 34 us; skew 500 ppm
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:03.318869 1608 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:04:03.319412 1608 webserver.cc:460] Webserver started at http://127.21.66.65:34301/ using document root <none> and password file <none>
I20260620 23:04:03.319687 1608 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:03.319794 1608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:03.319978 1608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:03.335420 1608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "f974da88c09a4465b054fb63e2e58b74"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "8b8ed04a9febde5b969ab7c702419d70"
server_key_iv: "86a1dfa2a9f22f801edf847f5ea62e11"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.335918 1608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "f974da88c09a4465b054fb63e2e58b74"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "8b8ed04a9febde5b969ab7c702419d70"
server_key_iv: "86a1dfa2a9f22f801edf847f5ea62e11"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.337910 1608 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 23:04:03.351948 1623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.352272 1608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:03.352391 1608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "f974da88c09a4465b054fb63e2e58b74"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "8b8ed04a9febde5b969ab7c702419d70"
server_key_iv: "86a1dfa2a9f22f801edf847f5ea62e11"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.352489 1608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:03.373225 1608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:03.375963 1608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:03.376199 1608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:03.376543 1608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:03.376947 1608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:03.377024 1608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.377084 1608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:03.377128 1608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.383510 1608 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:37397
I20260620 23:04:03.383989 1608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:03.394176 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 1608
I20260620 23:04:03.394292 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:04:03.394482 21769 external_mini_cluster.cc:1428] Setting key a1a4fa60b5c1f471bcb09ded286bb75a
I20260620 23:04:03.395314 1738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:37397 every 8 connection(s)
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:04:03.409269 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:03.396341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:57073 (local address 127.21.66.126:45919)
0620 23:04:03.396905 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:03.396907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:03.396916 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:03.399131 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:03.399133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:03.399151 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:03.399233 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:03.400097 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.400922 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.402520 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.402918 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.402966 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:03.405595 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:03.405597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:03.405598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:03.405606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:03.407550 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:03.408079 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:03.408080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:03.408081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:03.408146 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:03.408476 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:03.408477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:03.408477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:03.408623 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:03.408676 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:03.409056 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 23:04:03.409156 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":79,"threads_started":1}
I20260620 23:04:03.409835 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:03.395993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45919 (local address 127.21.66.65:57073)
0620 23:04:03.396263 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:03.396329 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:03.398908 (+ 2579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:03.399372 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:03.399377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:03.399652 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:03.399966 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:03.399978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.401050 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.401051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:03.402412 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:03.402416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.403079 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.403080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:03.403224 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:03.403849 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:03.403860 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:03.405476 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:03.407648 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:03.407656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:03.407659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:03.407987 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:03.408216 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:03.408216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:03.408216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:03.408401 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:03.408723 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:03.408729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:03.408899 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 23:04:03.409372 (+ 473us) client_negotiation.cc:240] Negotiation successful
0620 23:04:03.409581 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
I20260620 23:04:03.410494 1739 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45919
I20260620 23:04:03.410594 1739 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:03.410797 1739 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:03.411408 1526 ts_manager.cc:194] Registered new tserver with Master: f974da88c09a4465b054fb63e2e58b74 (127.21.66.65:37397)
I20260620 23:04:03.412349 1526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:57073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:04:03.430233 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45919
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:04:03.555042 1745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:03.555474 1745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:03.555560 1745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:03.557508 1745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:03.557650 1745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:03.557744 1745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:04:03.560281 1745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.66.84:34459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:03.560905 1745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:03.561246 1745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:03.568394 1752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:03.568394 1755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:03.568861 1753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:03.569141 1745 server_base.cc:1034] running on GCE node
I20260620 23:04:03.569362 1745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:03.569644 1745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:03.570789 1745 hybrid_clock.cc:648] HybridClock initialized: now 1781996643570758 us; error 43 us; skew 500 ppm
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:03.574151 1745 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:04:03.574775 1745 webserver.cc:460] Webserver started at http://127.21.66.66:40695/ using document root <none> and password file <none>
I20260620 23:04:03.575105 1745 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:03.575233 1745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:03.575421 1745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:03.591207 1745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "5a0ad8442e8d47c482f821d771e8007c"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "55f787813c199e93b818e4bc876e721b"
server_key_iv: "9fd0f59c6ec1cf12c1df8cb548c85987"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.591769 1745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "5a0ad8442e8d47c482f821d771e8007c"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "55f787813c199e93b818e4bc876e721b"
server_key_iv: "9fd0f59c6ec1cf12c1df8cb548c85987"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.594343 1745 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:04:03.618273 1762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.618575 1745 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260620 23:04:03.618697 1745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "5a0ad8442e8d47c482f821d771e8007c"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "55f787813c199e93b818e4bc876e721b"
server_key_iv: "9fd0f59c6ec1cf12c1df8cb548c85987"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.618835 1745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:03.631567 1745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:03.634016 1745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:03.634274 1745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:03.634574 1745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:03.634977 1745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:03.635085 1745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.635159 1745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:03.635232 1745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.642418 1745 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:38187
I20260620 23:04:03.642973 1745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:04:03.643114 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 1745
I20260620 23:04:03.643244 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:04:03.643471 21769 external_mini_cluster.cc:1428] Setting key 7fddadab1633b4b99232ce96ad445831
I20260620 23:04:03.658851 1876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:38187 every 8 connection(s)
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:03.671442 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:03.659190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:44175 (local address 127.21.66.126:45919)
0620 23:04:03.659241 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:03.659244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:03.659439 (+ 195us) server_negotiation.cc:407] Connection header received
0620 23:04:03.660213 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:03.660214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:03.660234 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:03.660267 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:03.661140 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.661903 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.663517 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.663834 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.663886 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:03.667726 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:03.667727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:03.667728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:03.667737 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:03.669617 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:03.670194 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:03.670196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:03.670196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:03.670257 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:03.670548 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:03.670549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:03.670549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:03.670712 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:03.670783 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:03.671190 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 23:04:03.671293 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:04:03.672290 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:03.659068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45919 (local address 127.21.66.66:44175)
0620 23:04:03.659302 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:03.659347 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:03.660076 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:03.660478 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:03.660482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:03.660721 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:03.661004 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:03.661011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.662029 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.662030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:03.663395 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:03.663398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.663994 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.663995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:03.664164 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:03.666014 (+ 1850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:03.666026 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:03.667610 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:03.669719 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:03.669727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:03.669731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:03.670075 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:03.670331 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:03.670331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:03.670332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:03.670455 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:03.670813 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:03.670820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:03.671016 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 23:04:03.671828 (+ 812us) client_negotiation.cc:240] Negotiation successful
0620 23:04:03.672026 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":119,"threads_started":1}
I20260620 23:04:03.673044 1877 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45919
I20260620 23:04:03.673168 1877 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:03.673358 1877 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:03.673763 1526 ts_manager.cc:194] Registered new tserver with Master: 5a0ad8442e8d47c482f821d771e8007c (127.21.66.66:38187)
I20260620 23:04:03.674302 1526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:44175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:04:03.685248 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45919
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:03.801050 1882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:03.801342 1882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:03.801381 1882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:03.803301 1882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:03.803364 1882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:03.803403 1882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:04:03.805579 1882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.66.84:34459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:03.805840 1882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:03.806161 1882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:03.812707 1890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:03.812777 1888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:03.813058 1887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:03.813452 1882 server_base.cc:1034] running on GCE node
I20260620 23:04:03.813753 1882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:03.814064 1882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:03.815268 1882 hybrid_clock.cc:648] HybridClock initialized: now 1781996643815246 us; error 35 us; skew 500 ppm
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:03.818554 1882 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:04:03.819094 1882 webserver.cc:460] Webserver started at http://127.21.66.67:44153/ using document root <none> and password file <none>
I20260620 23:04:03.819376 1882 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:03.819440 1882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:03.819573 1882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:03.834291 1882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "5402490175e24b64a06d77e3b4a266a5"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "591c03ddfcea15f4de4478319c14216a"
server_key_iv: "69968cf2974f4f9a94729235ac53f75d"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.834741 1882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "5402490175e24b64a06d77e3b4a266a5"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "591c03ddfcea15f4de4478319c14216a"
server_key_iv: "69968cf2974f4f9a94729235ac53f75d"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.836802 1882 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:03.855520 1897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.855747 1882 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 23:04:03.855842 1882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "5402490175e24b64a06d77e3b4a266a5"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "591c03ddfcea15f4de4478319c14216a"
server_key_iv: "69968cf2974f4f9a94729235ac53f75d"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:03.855950 1882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:03.890051 1882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:03.892660 1882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:03.892830 1882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:03.893121 1882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:03.893493 1882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:03.893544 1882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.893582 1882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:03.893601 1882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:03.900635 1882 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:43119
I20260620 23:04:03.901170 1882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:04:03.903260 2010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:43119 every 8 connection(s)
I20260620 23:04:03.906955 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 1882
I20260620 23:04:03.907083 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:04:03.907275 21769 external_mini_cluster.cc:1428] Setting key 733629f7d6c03fdef46e521bb63e0b40
Jun 20 23:04:03 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996643, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:03.913834 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:03.902232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:52527 (local address 127.21.66.126:45919)
0620 23:04:03.902279 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:03.902281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:03.902412 (+ 131us) server_negotiation.cc:407] Connection header received
0620 23:04:03.903169 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:03.903170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:03.903183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:03.903254 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:03.904171 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.905014 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.906634 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.906943 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.906989 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:03.910300 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:03.910302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:03.910302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:03.910310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:03.912038 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:03.912610 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:03.912611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:03.912611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:03.912680 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:03.913018 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:03.913019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:03.913019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:03.913174 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:03.913243 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:03.913636 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 23:04:03.913731 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:04:03.914529 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:03.901973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45919 (local address 127.21.66.67:52527)
0620 23:04:03.902260 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:03.902311 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:03.903031 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:03.903470 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:03.903475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:03.903743 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:03.904043 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:03.904049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.905141 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.905143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:03.906524 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:03.906527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:03.907116 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:03.907117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:03.907273 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:03.908409 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:03.908420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:03.910182 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:03.912138 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:03.912147 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:03.912150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:03.912496 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:03.912754 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:03.912755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:03.912755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:03.912917 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:03.913245 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:03.913252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:03.913436 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 23:04:03.913581 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 23:04:03.913759 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260620 23:04:03.914984 2011 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45919
I20260620 23:04:03.915112 2011 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:03.915318 2011 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:03.915737 1526 ts_manager.cc:194] Registered new tserver with Master: 5402490175e24b64a06d77e3b4a266a5 (127.21.66.67:43119)
I20260620 23:04:03.916303 1526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:52527
I20260620 23:04:03.919032 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:04:03.919943 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 1608
I20260620 23:04:03.926622 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 1745
I20260620 23:04:03.932116 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 1882
I20260620 23:04:03.938295 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 1494
I20260620 23:04:03.945266 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45919
--webserver_interface=127.21.66.126
--webserver_port=36181
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:45919
--ranger_config_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:04.029206 2014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:04.029418 2014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:04.029441 2014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:04.030719 2014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:04.030766 2014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:04.030781 2014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:04.030793 2014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:04.030805 2014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:04.032431 2014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:45919
--ranger_config_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:04.032681 2014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:04.032920 2014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:04.037904 2019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:04.037923 2022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:04.037959 2014 server_base.cc:1034] running on GCE node
W20260620 23:04:04.037922 2020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:04.038470 2014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:04.038782 2014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:04.039937 2014 hybrid_clock.cc:648] HybridClock initialized: now 1781996644039918 us; error 45 us; skew 500 ppm
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:04.042708 2014 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:04.043190 2014 webserver.cc:460] Webserver started at http://127.21.66.126:36181/ using document root <none> and password file <none>
I20260620 23:04:04.043378 2014 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:04.043426 2014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:04.119690 2014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:04.120438 2030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.120642 2014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:04.120754 2014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "8e9152ca98d5468e9659968477648533"
format_stamp: "Formatted at 2026-06-20 23:04:02 on dist-test-slave-qgtk"
server_key: "62625a5c4242a49bc2040db046664d40"
server_key_iv: "053fb01bd9cab13cfd441ea3278f2fa9"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:04.121238 2014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:04.137102 2014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:04.139348 2014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:04.139554 2014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:04.144667 2014 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:45919
I20260620 23:04:04.144726 2082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:45919 every 8 connection(s)
I20260620 23:04:04.145182 2014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:04.146565 2083 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:04:04.147172 2083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Bootstrap starting.
I20260620 23:04:04.152223 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2014
I20260620 23:04:04.152580 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:37397
--local_ip_for_outbound_sockets=127.21.66.65
--tserver_master_addrs=127.21.66.126:45919
--webserver_port=34301
--webserver_interface=127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:04.155354 2083 log.cc:824] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:04.157166 2083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:04:04.157464 2083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Bootstrap complete.
I20260620 23:04:04.159992 2083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:04.160274 2083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e9152ca98d5468e9659968477648533, State: Initialized, Role: FOLLOWER
I20260620 23:04:04.160491 2083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:04.160589 2083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:04.160645 2083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:04.160694 2083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:04:04.161743 2083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:04.161870 2083 leader_election.cc:302] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e9152ca98d5468e9659968477648533; no voters:
I20260620 23:04:04.162082 2083 leader_election.cc:288] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:04:04.162171 2087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:04:04.162320 2087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [term 2 LEADER]: Becoming Leader. State: Replica: 8e9152ca98d5468e9659968477648533, State: Running, Role: LEADER
I20260620 23:04:04.162393 2083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:04.162418 2087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } }
I20260620 23:04:04.162703 2087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e9152ca98d5468e9659968477648533. Latest consensus state: current_term: 2 leader_uuid: "8e9152ca98d5468e9659968477648533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } } }
I20260620 23:04:04.162776 2087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:04.162704 2089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8e9152ca98d5468e9659968477648533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e9152ca98d5468e9659968477648533" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45919 } } }
I20260620 23:04:04.162825 2089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:04.163179 2092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:04.163658 2092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:04.163805 2092 catalog_manager.cc:1259] Loaded cluster ID: 5dbc35f6990d47919958b6a9069a08ed
I20260620 23:04:04.163841 2092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:04.164075 2083 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:04:04.167949 2092 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:04.168499 2092 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Loaded TSK: 0
I20260620 23:04:04.168821 2092 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:04:04.264171 2085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:04.264407 2085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:04.264448 2085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:04.265651 2085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:04.265725 2085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:04.265774 2085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:04:04.267336 2085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:37397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=34301
--tserver_master_addrs=127.21.66.126:45919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:04.267624 2085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:04.267906 2085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:04.275588 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:04.275600 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:04.275638 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:04.276948 2085 server_base.cc:1034] running on GCE node
I20260620 23:04:04.277201 2085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:04.277482 2085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:04.278656 2085 hybrid_clock.cc:648] HybridClock initialized: now 1781996644278642 us; error 30 us; skew 500 ppm
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:04.282001 2085 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:04:04.282608 2085 webserver.cc:460] Webserver started at http://127.21.66.65:34301/ using document root <none> and password file <none>
I20260620 23:04:04.282897 2085 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:04.282990 2085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:04.303404 2085 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:04:04.306584 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.306758 2085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:04:04.306839 2085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "f974da88c09a4465b054fb63e2e58b74"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "8b8ed04a9febde5b969ab7c702419d70"
server_key_iv: "86a1dfa2a9f22f801edf847f5ea62e11"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:04.307308 2085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:04.338918 2085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:04.341212 2085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:04.341332 2085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:04.341537 2085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:04.341877 2085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:04.341935 2085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.341979 2085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:04.342026 2085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.349362 2085 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:37397
I20260620 23:04:04.349470 2244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:37397 every 8 connection(s)
I20260620 23:04:04.349992 2085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:04.351574 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2085
I20260620 23:04:04.351876 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:38187
--local_ip_for_outbound_sockets=127.21.66.66
--tserver_master_addrs=127.21.66.126:45919
--webserver_port=40695
--webserver_interface=127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:04.367185 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:04.351211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:34297 (local address 127.21.66.126:45919)
0620 23:04:04.354292 (+ 3081us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:04.354304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:04.354352 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:04:04.354513 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:04.354522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:04.354587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:04.355529 (+ 942us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:04.355629 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.357323 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.359095 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.359789 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.359908 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:04.362928 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:04.362933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:04.362933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:04.362948 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:04.365087 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.365587 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.365590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.365590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.365694 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.366041 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.366041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.366042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.366386 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:04.366469 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:04.366842 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 23:04:04.366975 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3008,"thread_start_us":107,"threads_started":1}
I20260620 23:04:04.367197 2247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:04.350894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45919 (local address 127.21.66.65:34297)
0620 23:04:04.351164 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:04.351209 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:04.351846 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:04.354812 (+ 2966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:04.354817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:04.355084 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:04.355382 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:04.355390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.357463 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.357464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:04.358959 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:04.358962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.359915 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.359916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:04.360084 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:04.360763 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:04.360773 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:04.362791 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:04.365192 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:04.365199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:04.365201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:04.365494 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:04.365788 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:04.365790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:04.365790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:04.365942 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:04.366468 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:04.366475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:04.366651 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 23:04:04.366778 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:04:04.366947 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":101,"threads_started":1}
I20260620 23:04:04.368446 2245 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45919
I20260620 23:04:04.368561 2245 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.368806 2245 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
W20260620 23:04:04.369799 2245 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:45919 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:04:04.369886 2245 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.369999 2245 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:04.370395 2245 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.370524 2245 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
W20260620 23:04:04.370865 2245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:04:04.458252 2249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:04.458630 2249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:04.458722 2249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:04.460568 2249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:04.460636 2249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:04.460677 2249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:04:04.462986 2249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:38187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=40695
--tserver_master_addrs=127.21.66.126:45919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:04.463399 2249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:04.463702 2249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:04.469525 2255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:04.469595 2249 server_base.cc:1034] running on GCE node
W20260620 23:04:04.469741 2257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:04.469852 2254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:04.470244 2249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:04.470587 2249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:04.471774 2249 hybrid_clock.cc:648] HybridClock initialized: now 1781996644471754 us; error 35 us; skew 500 ppm
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:04.474563 2249 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:04:04.475206 2249 webserver.cc:460] Webserver started at http://127.21.66.66:40695/ using document root <none> and password file <none>
I20260620 23:04:04.475453 2249 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:04.475545 2249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:04.495491 2249 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260620 23:04:04.498056 2265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.498363 2249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:04:04.498493 2249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "5a0ad8442e8d47c482f821d771e8007c"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "55f787813c199e93b818e4bc876e721b"
server_key_iv: "9fd0f59c6ec1cf12c1df8cb548c85987"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:04.498975 2249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:04.516348 2249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:04.519729 2249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:04.519945 2249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:04.520233 2249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:04.520684 2249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:04.520846 2249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.520915 2249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:04.520970 2249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.530042 2249 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:38187
I20260620 23:04:04.530598 2249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:04:04.531082 2378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:38187 every 8 connection(s)
I20260620 23:04:04.534910 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2249
I20260620 23:04:04.535197 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.66.84:34459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:43119
--local_ip_for_outbound_sockets=127.21.66.67
--tserver_master_addrs=127.21.66.126:45919
--webserver_port=44153
--webserver_interface=127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:04.548135 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:04.531522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:33177 (local address 127.21.66.126:45919)
0620 23:04:04.531577 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:04.531580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:04.532287 (+ 707us) server_negotiation.cc:407] Connection header received
0620 23:04:04.533010 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:04.533012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:04.533027 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:04.533088 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:04.533841 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.534461 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.536068 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.536417 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.536516 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:04.544306 (+ 7790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:04.544308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:04.544309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:04.544318 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:04.546332 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.546886 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.546888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.546888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.546946 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.547256 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.547257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.547258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.547407 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:04.547451 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:04.547929 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 23:04:04.548032 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:04:04.548395 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:04.531541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45919 (local address 127.21.66.66:33177)
0620 23:04:04.532132 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:04.532190 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:04.532869 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:04.533172 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:04.533177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:04.533436 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:04.533730 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:04.533737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.534651 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.534652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:04.535967 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:04.535970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.536922 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.536923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:04.537101 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:04.537707 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:04.537724 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:04.544126 (+ 6402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:04.546436 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:04.546447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:04.546449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:04.546794 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:04.547014 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:04.547014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:04.547015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:04.547171 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:04.547506 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:04.547513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:04.547695 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 23:04:04.547870 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 23:04:04.548076 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":422,"threads_started":1}
I20260620 23:04:04.548985 2379 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45919
I20260620 23:04:04.549106 2379 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.549311 2379 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
W20260620 23:04:04.549952 2379 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:45919 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:04:04.550040 2379 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.550109 2379 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:04.550436 2379 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.550518 2379 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
W20260620 23:04:04.550859 2379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:04:04.640039 2382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:04.640324 2382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:04.640362 2382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:04.642151 2382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:04.642207 2382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:04.642247 2382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:04:04.644523 2382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.66.84:34459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:43119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=44153
--tserver_master_addrs=127.21.66.126:45919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:04.644773 2382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:04.645040 2382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:04.651576 2388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:04.651576 2387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:04.651975 2390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:04.652716 2382 server_base.cc:1034] running on GCE node
I20260620 23:04:04.652941 2382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:04.653219 2382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:04.654397 2382 hybrid_clock.cc:648] HybridClock initialized: now 1781996644654363 us; error 46 us; skew 500 ppm
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:04.657859 2382 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:04:04.658515 2382 webserver.cc:460] Webserver started at http://127.21.66.67:44153/ using document root <none> and password file <none>
I20260620 23:04:04.658818 2382 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:04.658903 2382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:04.674538 2382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:04:04.678627 2399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.678824 2382 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 23:04:04.678931 2382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "5402490175e24b64a06d77e3b4a266a5"
format_stamp: "Formatted at 2026-06-20 23:04:03 on dist-test-slave-qgtk"
server_key: "591c03ddfcea15f4de4478319c14216a"
server_key_iv: "69968cf2974f4f9a94729235ac53f75d"
server_key_version: "kuduclusterkey@0"
I20260620 23:04:04.679426 2382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:04.690584 2382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:04.692921 2382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:04.693117 2382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:04.693408 2382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:04.693802 2382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:04.693859 2382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.693894 2382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:04.693917 2382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:04.701468 2382 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:43119
I20260620 23:04:04.701555 2512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:43119 every 8 connection(s)
I20260620 23:04:04.701892 2382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:04:04 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996644, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:04.712015 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2382
I20260620 23:04:04.712610 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:04.702707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:50361 (local address 127.21.66.126:45919)
0620 23:04:04.702759 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:04.702762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:04.702967 (+ 205us) server_negotiation.cc:407] Connection header received
0620 23:04:04.703599 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:04.703600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:04.703613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:04.703665 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:04.704314 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.704804 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.706192 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.706482 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.706564 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:04.709127 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:04.709128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:04.709129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:04.709138 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:04.710957 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.711485 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.711486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.711487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.711559 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.711843 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.711844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.711845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.711968 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:04.712055 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:04.712393 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 23:04:04.712494 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:04:04.713295 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:04.702665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45919 (local address 127.21.66.67:50361)
0620 23:04:04.702824 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:04.702868 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:04.703467 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:04.703770 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:04.703774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:04.704017 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:04.704221 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:04.704225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.704905 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.704906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:04.705732 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:04.705734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.706618 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.706619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:04.706780 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:04.707440 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:04.707450 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:04.709008 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:04.711065 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:04.711072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:04.711074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:04.711384 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:04.711629 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:04.711630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:04.711630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:04.711772 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:04.712045 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:04.712051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:04.712204 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 23:04:04.712832 (+ 628us) client_negotiation.cc:240] Negotiation successful
0620 23:04:04.713008 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260620 23:04:04.713842 2513 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45919
I20260620 23:04:04.713941 2513 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.714160 2513 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
W20260620 23:04:04.714880 2513 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:45919 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:04:04.714957 2513 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.715015 2513 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:04.715304 2513 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:04.715409 2513 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
W20260620 23:04:04.715714 2513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:04:04.720181 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:04.712418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.21.66.126:45919)
0620 23:04:04.712695 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:04.712697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:04.712705 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:04:04.712726 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:04.712727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:04.712737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:04.712789 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:04.713675 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.714393 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.715497 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:04.715791 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:04.715849 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:04.717394 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:04.717395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:04.717396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:04.717404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:04.718985 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.719398 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.719399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.719399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.719446 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:04.719656 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:04.719657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:04.719657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:04.719755 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:04.719873 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:04.720008 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 23:04:04.720086 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":64,"threads_started":1}
Jun 20 23:04:05 dist-test-slave-qgtk krb5kdc[1050](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781996645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:05.371137 2245 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:05.371301 2245 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:05.371686 2047 ts_manager.cc:194] Registered new tserver with Master: f974da88c09a4465b054fb63e2e58b74 (127.21.66.65:37397)
I20260620 23:04:05.372689 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:34297
I20260620 23:04:05.551055 2379 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:05.551235 2379 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:05.551620 2047 ts_manager.cc:194] Registered new tserver with Master: 5a0ad8442e8d47c482f821d771e8007c (127.21.66.66:38187)
I20260620 23:04:05.552165 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:33177
Jun 20 23:04:05 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
I20260620 23:04:05.715895 2513 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:05.716058 2513 heartbeater.cc:511] Master 127.21.66.126:45919 requested a full tablet report, sending...
I20260620 23:04:05.716439 2047 ts_manager.cc:194] Registered new tserver with Master: 5402490175e24b64a06d77e3b4a266a5 (127.21.66.67:43119)
I20260620 23:04:05.716931 2047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:50361
I20260620 23:04:05.721493 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:04:05.733039 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.722564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.21.66.126:45919)
0620 23:04:05.722693 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:05.722696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:05.724981 (+ 2285us) server_negotiation.cc:407] Connection header received
0620 23:04:05.725015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:05.725016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:05.725030 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:05.725111 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:05.725918 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.726771 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.727846 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.728168 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.728244 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.729774 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:05.729775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:05.729775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:05.729784 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:05.731721 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:05.732129 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:05.732130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:05.732130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:05.732203 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:05.732420 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:05.732421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:05.732421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:05.732569 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:05.732680 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:05.732825 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 23:04:05.732925 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260620 23:04:05.735709 2047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:04:05.738025 2047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:04:05.750298 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.741513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43119 (local address 127.0.0.1:38798)
0620 23:04:05.745785 (+ 4272us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:05.745792 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:05.745851 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:05.746164 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:05.746168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:05.746170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:05.746250 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.746252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.747545 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.747547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.748889 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.748891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.749920 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.749921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.750039 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.750052 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:04:05.750117 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:04:05.750184 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4260,"thread_start_us":100,"threads_started":1}
I20260620 23:04:05.750758 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.741796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.21.66.67:43119)
0620 23:04:05.745748 (+ 3952us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:05.745752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:05.745849 (+ 97us) server_negotiation.cc:407] Connection header received
0620 23:04:05.745915 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:05.745918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:05.745935 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:05.746025 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:05.746333 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.747446 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.748992 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.749770 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.749835 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.749889 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:05.750636 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 23:04:05.750681 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3921,"thread_start_us":51,"threads_started":1}
I20260620 23:04:05.751559 2541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.741505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37397 (local address 127.0.0.1:47028)
0620 23:04:05.745531 (+ 4026us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:05.745579 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:05.745637 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:05.746273 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:05.746273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:05.746275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:05.746321 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.746323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.749106 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.749107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.750193 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.750195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.751345 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.751346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.751407 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.751411 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:04:05.751454 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 23:04:05.751488 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3986,"thread_start_us":102,"threads_started":1}
I20260620 23:04:05.752089 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.741796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.21.66.65:37397)
0620 23:04:05.746091 (+ 4295us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:05.746096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:05.746106 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:04:05.746159 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:05.746161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:05.746176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:05.746396 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:05.746487 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.747481 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.750283 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.751257 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.751876 (+ 619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.751949 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:05.751973 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 23:04:05.752014 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4259,"thread_start_us":59,"threads_started":1}
I20260620 23:04:05.752269 2442 tablet_service.cc:1467] Processing CreateTablet for tablet 5ceac7a1974c4f80a9c434eb2e1879b7 (DEFAULT_TABLE table=test-table [id=2998cbeb7208420399e10e8be599ca09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:05.752689 2442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ceac7a1974c4f80a9c434eb2e1879b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:05.752825 2179 tablet_service.cc:1467] Processing CreateTablet for tablet 5ceac7a1974c4f80a9c434eb2e1879b7 (DEFAULT_TABLE table=test-table [id=2998cbeb7208420399e10e8be599ca09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:05.753159 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ceac7a1974c4f80a9c434eb2e1879b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:05.753314 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.741868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:38187 (local address 127.0.0.1:38090)
0620 23:04:05.745932 (+ 4064us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:05.745940 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:05.745977 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:05.747795 (+ 1818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:05.747796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:05.747798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:05.747833 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.747834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.751065 (+ 3231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.751067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.752148 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.752152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.753105 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.753107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.753170 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.753174 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:04:05.753206 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 23:04:05.753244 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4050,"thread_start_us":74,"threads_started":1}
I20260620 23:04:05.753628 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.742036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.21.66.66:38187)
0620 23:04:05.747614 (+ 5578us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:05.747618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:05.747627 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:05.747660 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:05.747662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:05.747702 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:05.747899 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:05.747987 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.749007 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.752245 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.753005 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.753405 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.753474 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:05.753500 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 23:04:05.753544 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5539,"thread_start_us":114,"threads_started":1}
I20260620 23:04:05.755919 2546 tablet_bootstrap.cc:492] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74: Bootstrap starting.
I20260620 23:04:05.756572 2546 tablet_bootstrap.cc:654] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:05.756918 2546 log.cc:824] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:05.757340 2547 tablet_bootstrap.cc:492] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Bootstrap starting.
I20260620 23:04:05.757531 2310 tablet_service.cc:1467] Processing CreateTablet for tablet 5ceac7a1974c4f80a9c434eb2e1879b7 (DEFAULT_TABLE table=test-table [id=2998cbeb7208420399e10e8be599ca09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:05.757867 2310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ceac7a1974c4f80a9c434eb2e1879b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:05.757964 2547 tablet_bootstrap.cc:654] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:05.758302 2547 log.cc:824] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:05.759049 2546 tablet_bootstrap.cc:492] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74: No bootstrap required, opened a new log
I20260620 23:04:05.759107 2546 ts_tablet_manager.cc:1397] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 23:04:05.761906 2546 raft_consensus.cc:348] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.762095 2546 raft_consensus.cc:374] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:05.762163 2546 raft_consensus.cc:729] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f974da88c09a4465b054fb63e2e58b74, State: Initialized, Role: FOLLOWER
I20260620 23:04:05.762272 2546 consensus_queue.cc:260] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.762534 2546 ts_tablet_manager.cc:1428] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 23:04:05.762547 2245 heartbeater.cc:503] Master 127.21.66.126:45919 was elected leader, sending a full tablet report...
I20260620 23:04:05.763345 2547 tablet_bootstrap.cc:492] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: No bootstrap required, opened a new log
I20260620 23:04:05.763420 2547 ts_tablet_manager.cc:1397] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 23:04:05.765542 2547 raft_consensus.cc:348] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.765705 2547 raft_consensus.cc:374] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:05.765748 2547 raft_consensus.cc:729] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5402490175e24b64a06d77e3b4a266a5, State: Initialized, Role: FOLLOWER
I20260620 23:04:05.765872 2547 consensus_queue.cc:260] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.766198 2513 heartbeater.cc:503] Master 127.21.66.126:45919 was elected leader, sending a full tablet report...
I20260620 23:04:05.766239 2547 ts_tablet_manager.cc:1428] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:04:05.766427 2550 tablet_bootstrap.cc:492] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Bootstrap starting.
I20260620 23:04:05.767143 2550 tablet_bootstrap.cc:654] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:05.767475 2550 log.cc:824] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:05.768213 2550 tablet_bootstrap.cc:492] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: No bootstrap required, opened a new log
I20260620 23:04:05.768304 2550 ts_tablet_manager.cc:1397] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:04:05.770098 2550 raft_consensus.cc:348] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.770320 2550 raft_consensus.cc:374] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:05.770367 2550 raft_consensus.cc:729] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0ad8442e8d47c482f821d771e8007c, State: Initialized, Role: FOLLOWER
I20260620 23:04:05.770499 2550 consensus_queue.cc:260] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.770773 2550 ts_tablet_manager.cc:1428] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:04:05.770807 2379 heartbeater.cc:503] Master 127.21.66.126:45919 was elected leader, sending a full tablet report...
W20260620 23:04:05.783077 2380 tablet.cc:2369] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:04:05.851573 2246 tablet.cc:2369] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:04:05 dist-test-slave-qgtk krb5kdc[1050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996645, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for HTTP/127.21.66.84@KRBTEST.COM
W20260620 23:04:05.953413 2514 tablet.cc:2369] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:04:05.984514 2554 raft_consensus.cc:482] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:05.984685 2554 raft_consensus.cc:504] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.985131 2554 leader_election.cc:288] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f974da88c09a4465b054fb63e2e58b74 (127.21.66.65:37397), 5402490175e24b64a06d77e3b4a266a5 (127.21.66.67:43119)
I20260620 23:04:05.992144 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.985789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:41273 (local address 127.21.66.65:37397)
0620 23:04:05.985852 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:05.985854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:05.986219 (+ 365us) server_negotiation.cc:407] Connection header received
0620 23:04:05.986353 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:05.986355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:05.986371 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:05.986405 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:05.986794 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.987660 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.991018 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.991678 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.991736 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.991746 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:05.992010 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:04:05.992060 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:04:05.993851 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.985435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37397 (local address 127.21.66.66:41273)
0620 23:04:05.986097 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:05.986160 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:05.986258 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:05.986611 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:05.986612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:05.986615 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:05.986706 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.986708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.987833 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.987835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.990883 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.990890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.991835 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.991837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.991914 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.991918 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:04:05.992524 (+ 606us) client_negotiation.cc:240] Negotiation successful
0620 23:04:05.992586 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":116,"threads_started":1}
I20260620 23:04:05.994432 2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceac7a1974c4f80a9c434eb2e1879b7" candidate_uuid: "5a0ad8442e8d47c482f821d771e8007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f974da88c09a4465b054fb63e2e58b74" is_pre_election: true
I20260620 23:04:05.994611 2199 raft_consensus.cc:2454] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a0ad8442e8d47c482f821d771e8007c in term 0.
I20260620 23:04:05.994974 2268 leader_election.cc:302] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0ad8442e8d47c482f821d771e8007c, f974da88c09a4465b054fb63e2e58b74; no voters:
I20260620 23:04:05.995129 2554 raft_consensus.cc:2781] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:04:05.995182 2554 raft_consensus.cc:482] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:04:05.995204 2554 raft_consensus.cc:3037] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:05.996206 2554 raft_consensus.cc:504] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.996362 2554 leader_election.cc:288] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [CANDIDATE]: Term 1 election: Requested vote from peers f974da88c09a4465b054fb63e2e58b74 (127.21.66.65:37397), 5402490175e24b64a06d77e3b4a266a5 (127.21.66.67:43119)
I20260620 23:04:05.996745 2199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceac7a1974c4f80a9c434eb2e1879b7" candidate_uuid: "5a0ad8442e8d47c482f821d771e8007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f974da88c09a4465b054fb63e2e58b74"
I20260620 23:04:05.996831 2199 raft_consensus.cc:3037] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:05.997210 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.986212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:48379 (local address 127.21.66.67:43119)
0620 23:04:05.986262 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:05.986264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:05.987937 (+ 1673us) server_negotiation.cc:407] Connection header received
0620 23:04:05.989011 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:05.989013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:05.989025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:05.989112 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:05.989294 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.990043 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.996168 (+ 6125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.996753 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.996806 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.996815 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:05.997090 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 23:04:05.997137 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:04:05.997450 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:05.985875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43119 (local address 127.21.66.66:48379)
0620 23:04:05.987863 (+ 1988us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:05.987880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:05.987967 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:05.989116 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:05.989117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:05.989133 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:05.989202 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.989203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.992676 (+ 3473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.992677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.996053 (+ 3376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:05.996057 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:05.996881 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:05.996882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:05.996967 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:05.996972 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:05.997325 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 23:04:05.997370 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"thread_start_us":100,"threads_started":1}
I20260620 23:04:05.997705 2199 raft_consensus.cc:2454] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0ad8442e8d47c482f821d771e8007c in term 1.
I20260620 23:04:05.997798 2466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceac7a1974c4f80a9c434eb2e1879b7" candidate_uuid: "5a0ad8442e8d47c482f821d771e8007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5402490175e24b64a06d77e3b4a266a5"
I20260620 23:04:05.997884 2268 leader_election.cc:302] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a0ad8442e8d47c482f821d771e8007c, f974da88c09a4465b054fb63e2e58b74; no voters:
I20260620 23:04:05.997876 2467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ceac7a1974c4f80a9c434eb2e1879b7" candidate_uuid: "5a0ad8442e8d47c482f821d771e8007c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5402490175e24b64a06d77e3b4a266a5" is_pre_election: true
I20260620 23:04:05.997920 2466 raft_consensus.cc:3037] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:05.998466 2554 raft_consensus.cc:2781] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:05.998646 2554 raft_consensus.cc:686] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 1 LEADER]: Becoming Leader. State: Replica: 5a0ad8442e8d47c482f821d771e8007c, State: Running, Role: LEADER
I20260620 23:04:05.998744 2554 consensus_queue.cc:237] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } }
I20260620 23:04:05.998847 2466 raft_consensus.cc:2454] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a0ad8442e8d47c482f821d771e8007c in term 1.
I20260620 23:04:05.999770 2047 catalog_manager.cc:5495] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a0ad8442e8d47c482f821d771e8007c (127.21.66.66). New cstate: current_term: 1 leader_uuid: "5a0ad8442e8d47c482f821d771e8007c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a0ad8442e8d47c482f821d771e8007c" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 } health_report { overall_health: UNKNOWN } } }
I20260620 23:04:06.044170 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:06.040530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.21.66.66:38187)
0620 23:04:06.040602 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:06.040604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:06.040614 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:04:06.040659 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:06.040660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:06.040682 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:06.040757 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:04:06.040942 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:06.041734 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:06.042989 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:06.043322 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:06.043383 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:06.043599 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:04:06.043736 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:04:06.043813 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:06.044064 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 23:04:06.044086 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:04:06.046656 2466 raft_consensus.cc:1261] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 1 FOLLOWER]: Refusing update from remote peer 5a0ad8442e8d47c482f821d771e8007c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:06.046811 2199 raft_consensus.cc:1261] T 5ceac7a1974c4f80a9c434eb2e1879b7 P f974da88c09a4465b054fb63e2e58b74 [term 1 FOLLOWER]: Refusing update from remote peer 5a0ad8442e8d47c482f821d771e8007c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:06.047142 2554 consensus_queue.cc:1035] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5402490175e24b64a06d77e3b4a266a5" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:06.048831 2559 consensus_queue.cc:1035] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f974da88c09a4465b054fb63e2e58b74" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:06.052237 2562 mvcc.cc:205] Tried to move back new op lower bound from 7299058262204403712 to 7299058262011404288. Current Snapshot: MvccSnapshot[applied={T|T < 7299058262204403712}]
I20260620 23:04:06.052445 2563 mvcc.cc:205] Tried to move back new op lower bound from 7299058262204403712 to 7299058262011404288. Current Snapshot: MvccSnapshot[applied={T|T < 7299058262204403712}]
I20260620 23:04:06.055583 2047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:06.055698 2047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:06.056689 2047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e9152ca98d5468e9659968477648533: Sending DeleteTablet for 3 replicas of tablet 5ceac7a1974c4f80a9c434eb2e1879b7
I20260620 23:04:06.057119 2179 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ceac7a1974c4f80a9c434eb2e1879b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:06 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:47028
I20260620 23:04:06.057155 2442 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ceac7a1974c4f80a9c434eb2e1879b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:06 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:38798
I20260620 23:04:06.057657 2310 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ceac7a1974c4f80a9c434eb2e1879b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:06 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:38090
I20260620 23:04:06.058135 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2085
I20260620 23:04:06.061843 2571 tablet_replica.cc:330] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: stopping tablet replica
I20260620 23:04:06.062060 2571 raft_consensus.cc:2229] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:06.062317 2571 raft_consensus.cc:2258] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:06.062634 2572 tablet_replica.cc:330] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: stopping tablet replica
I20260620 23:04:06.062824 2572 raft_consensus.cc:2229] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:04:06.062978 2572 raft_consensus.cc:2258] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:06.064110 2572 ts_tablet_manager.cc:1905] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:06.066177 2571 ts_tablet_manager.cc:1905] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 23:04:06.066738 2268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:37397 (error 108)
I20260620 23:04:06.071583 2571 ts_tablet_manager.cc:1918] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:06.071810 2571 log.cc:1196] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/5ceac7a1974c4f80a9c434eb2e1879b7
I20260620 23:04:06.071959 2572 ts_tablet_manager.cc:1918] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:06.072186 2571 ts_tablet_manager.cc:1939] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5402490175e24b64a06d77e3b4a266a5: Deleting consensus metadata
I20260620 23:04:06.072647 2572 log.cc:1196] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/5ceac7a1974c4f80a9c434eb2e1879b7
I20260620 23:04:06.072762 2034 catalog_manager.cc:4841] TS 5402490175e24b64a06d77e3b4a266a5 (127.21.66.67:43119): tablet 5ceac7a1974c4f80a9c434eb2e1879b7 (table test-table [id=2998cbeb7208420399e10e8be599ca09]) successfully deleted
W20260620 23:04:06.073174 2033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:37397 (error 108)
I20260620 23:04:06.074745 2572 ts_tablet_manager.cc:1939] T 5ceac7a1974c4f80a9c434eb2e1879b7 P 5a0ad8442e8d47c482f821d771e8007c: Deleting consensus metadata
I20260620 23:04:06.076979 2031 catalog_manager.cc:4841] TS 5a0ad8442e8d47c482f821d771e8007c (127.21.66.66:38187): tablet 5ceac7a1974c4f80a9c434eb2e1879b7 (table test-table [id=2998cbeb7208420399e10e8be599ca09]) successfully deleted
I20260620 23:04:06.078537 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:06.078396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37397 (local address 127.21.66.66:44695)
0620 23:04:06.078463 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:06.078500 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 23:04:06.078711 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:06.078401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37397 (local address 127.0.0.1:47032)
0620 23:04:06.078631 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:06.078677 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
W20260620 23:04:06.078907 2033 catalog_manager.cc:4584] TS f974da88c09a4465b054fb63e2e58b74 (127.21.66.65:37397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ceac7a1974c4f80a9c434eb2e1879b7: Network error: Client connection negotiation failed: client connection to 127.21.66.65:37397: connect: Connection refused (error 111)
I20260620 23:04:06.082453 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2249
I20260620 23:04:06.089395 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2382
I20260620 23:04:06.097512 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2014
I20260620 23:04:06.104342 21769 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:04:06.449447 21769 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:04:06.449657 21769 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:04:06.794605 21769 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:04:06.794 UTC [1080] LOG: received smart shutdown request
2026-06-20 23:04:06.798 UTC [1080] LOG: background worker "logical replication launcher" (PID 1089) exited with exit code 1
2026-06-20 23:04:06.799 UTC [1084] LOG: shutting down
2026-06-20 23:04:06.997 UTC [1080] LOG: database system is shut down
2026-06-20T23:04:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51169 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:04:07 dist-test-slave-qgtk krb5kdc[2583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:04:07 dist-test-slave-qgtk krb5kdc[2583](info): set up 2 sockets
Jun 20 23:04:07 dist-test-slave-qgtk krb5kdc[2583](info): commencing operation
krb5kdc: starting...
W20260620 23:04:08.214385 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:04:08.341182 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:38881
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:38881,127.21.66.125:39937,127.21.66.124:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:08.426905 2596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:08.427129 2596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:08.427152 2596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:08.428639 2596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:08.428692 2596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:08.428707 2596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:08.428721 2596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:08.428733 2596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:08.430511 2596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:38881,127.21.66.125:39937,127.21.66.124:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:38881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:08.430791 2596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:08.431056 2596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:08.436182 2601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:08.436188 2602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:08.436188 2604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:08.436578 2596 server_base.cc:1034] running on GCE node
I20260620 23:04:08.436928 2596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:08.437290 2596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:08.438484 2596 hybrid_clock.cc:648] HybridClock initialized: now 1781996648438458 us; error 66 us; skew 500 ppm
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:08.444696 2596 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:08.445243 2596 webserver.cc:460] Webserver started at http://127.21.66.126:41571/ using document root <none> and password file <none>
I20260620 23:04:08.445554 2596 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:08.445634 2596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:08.445799 2596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:08.447034 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "4aaaef84f99b4a29aa85756c5deef09f"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "f0e3798b3bb445a9c1cd0470d107dff0"
server_key_iv: "280d1e3e7a90c66bac9bdcada2c3dec4"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.447386 2596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "4aaaef84f99b4a29aa85756c5deef09f"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "f0e3798b3bb445a9c1cd0470d107dff0"
server_key_iv: "280d1e3e7a90c66bac9bdcada2c3dec4"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.449038 2596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:04:08.450052 2611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:08.450253 2596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:08.450363 2596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "4aaaef84f99b4a29aa85756c5deef09f"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "f0e3798b3bb445a9c1cd0470d107dff0"
server_key_iv: "280d1e3e7a90c66bac9bdcada2c3dec4"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.450464 2596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:08.474016 2596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:08.476604 2596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:08.476861 2596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:08.481444 2596 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:38881
I20260620 23:04:08.481513 2663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:38881 every 8 connection(s)
I20260620 23:04:08.481886 2596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:08.482784 2664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:08.484421 2664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:04:08.486995 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2596
I20260620 23:04:08.487087 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:04:08.487275 21769 external_mini_cluster.cc:1428] Setting key dac953a1119e6f83ebe72e5afb2df5da
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.125@KRBTEST.COM; defaulting to no policy
I20260620 23:04:08.496594 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.485200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.21.66.126:38881)
0620 23:04:08.485402 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.485406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.485495 (+ 89us) server_negotiation.cc:407] Connection header received
0620 23:04:08.486016 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.486018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.486035 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.486164 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.486878 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.487795 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.489041 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.489657 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.489768 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.492421 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.492429 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.492430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.492438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.494712 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.495194 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.495197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.495198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.495343 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.495583 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.495585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.495585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.495811 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.495906 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.496124 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.496319 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260620 23:04:08.496596 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.485153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:38881 (local address 127.0.0.1:52650)
0620 23:04:08.485323 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.485370 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.485861 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.486243 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.486245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.486570 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.486728 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.486733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.487896 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.487897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.488946 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.488948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.489773 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.489774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.489848 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.490590 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.490600 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.492322 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.494841 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.494848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.494850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.495070 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.495420 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.495421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.495422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.495512 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.495882 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.495887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.495950 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.496063 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.496338 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":129,"thread_start_us":72,"threads_started":1}
I20260620 23:04:08.497437 2664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:04:08.497879 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.497748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39274)
0620 23:04:08.497804 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.497852 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.125@KRBTEST.COM" created.
W20260620 23:04:08.498060 2613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
I20260620 23:04:08.498528 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.498423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39284)
0620 23:04:08.498473 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.498501 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 23:04:08.498687 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:39937: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab.
Entry for principal kudu/127.21.66.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab.
I20260620 23:04:08.515249 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:39937
--webserver_interface=127.21.66.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab
--principal=kudu/127.21.66.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:38881,127.21.66.125:39937,127.21.66.124:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:08.547951 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } attempt: 1
I20260620 23:04:08.548519 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.548366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39290)
0620 23:04:08.548442 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.548489 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:04:08.548990 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.548829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39304)
0620 23:04:08.548910 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.548953 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:04:08.549121 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:39937: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
W20260620 23:04:08.596861 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:08.597139 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:08.597179 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:08.598371 2671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:08.598438 2671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:08.598465 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:08.598485 2671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:08.598518 2671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:08.600029 2671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:38881,127.21.66.125:39937,127.21.66.124:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab
--principal=kudu/127.21.66.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.66.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:08.600307 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:08.600539 2671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:08.605636 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:08.605667 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:08.605881 2671 server_base.cc:1034] running on GCE node
W20260620 23:04:08.605681 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:08.606290 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:08.606659 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:08.607841 2671 hybrid_clock.cc:648] HybridClock initialized: now 1781996648607826 us; error 41 us; skew 500 ppm
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:08.610654 2671 init.cc:377] Logged in from keytab as kudu/127.21.66.125@KRBTEST.COM (short username kudu)
I20260620 23:04:08.611346 2671 webserver.cc:460] Webserver started at http://127.21.66.125:34259/ using document root <none> and password file <none>
I20260620 23:04:08.611595 2671 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:08.611670 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:08.611841 2671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:08.613003 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data/instance:
uuid: "9493b2dff88e4e93b6af94e0a81a750c"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "54af1babcd06b50862da183f80de891b"
server_key_iv: "e046826beff2b390975172f925bf662c"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.613394 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/wal/instance:
uuid: "9493b2dff88e4e93b6af94e0a81a750c"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "54af1babcd06b50862da183f80de891b"
server_key_iv: "e046826beff2b390975172f925bf662c"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.614898 2671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:04:08.615729 2686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:08.615993 2671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:08.616099 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/wal
uuid: "9493b2dff88e4e93b6af94e0a81a750c"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "54af1babcd06b50862da183f80de891b"
server_key_iv: "e046826beff2b390975172f925bf662c"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.616189 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:08.617317 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } attempt: 2
I20260620 23:04:08.617725 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.617588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39306)
0620 23:04:08.617656 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.617695 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:04:08.618083 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.617967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39310)
0620 23:04:08.618030 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.618059 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:04:08.618194 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:39937: Network error: Client connection negotiation failed: client connection to 127.21.66.125:39937: connect: Connection refused (error 111)
I20260620 23:04:08.626605 2671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:08.628569 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:08.628744 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:08.633265 2671 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.125:39937
I20260620 23:04:08.633402 2738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.125:39937 every 8 connection(s)
I20260620 23:04:08.633713 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
I20260620 23:04:08.634729 2739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:08.636327 2739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:04:08.640496 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2671
I20260620 23:04:08.640638 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-1/wal/instance
I20260620 23:04:08.640829 21769 external_mini_cluster.cc:1428] Setting key 7e853181e72c9f2248f03215aaf4a331
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:08.645990 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.637059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.21.66.126:38881)
0620 23:04:08.637155 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.637158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.637353 (+ 195us) server_negotiation.cc:407] Connection header received
0620 23:04:08.637811 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.637813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.637826 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.637891 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.638589 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.639107 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.640158 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.640407 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.640470 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.642992 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.642993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.642994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.643002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.644449 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.644989 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.644990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.644991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.645055 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.645329 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.645330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.645330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.645453 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.645531 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.645804 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.645899 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:04:08.646266 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.637012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:38881 (local address 127.0.0.1:52658)
0620 23:04:08.637194 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.637228 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.637693 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.637958 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.637961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.638301 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.638501 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.638506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.639205 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.639205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.640059 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.640061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.640498 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.640499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.640666 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.641237 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.641248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.642892 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.644641 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.644649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.644651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.644895 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.645124 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.645125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.645126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.645259 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.645522 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.645529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.645669 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.645781 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.645942 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260620 23:04:08.646842 2739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.66.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.124@KRBTEST.COM" created.
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:04:08.656230 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.647174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.21.66.125:39937)
0620 23:04:08.647326 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.647330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.647350 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:04:08.647374 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.647375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.647386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.647463 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.647726 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.648478 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.649409 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.649720 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.649969 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.652624 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.652636 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.652637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.652647 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.654897 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.655363 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.655366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.655366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.655426 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.655639 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.655640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.655640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.655799 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.655896 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.656051 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.656132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260620 23:04:08.656349 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.647129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39324)
0620 23:04:08.647180 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.647191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.647232 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.647463 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.647464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.647605 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.647646 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.647648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.648713 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.648714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.649254 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.649256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.649968 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.649969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.650040 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.650487 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.650491 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.652446 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.654997 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.654998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.654999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.655233 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.655499 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.655500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.655500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.655574 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.655876 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.655877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.655932 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.656022 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.656200 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":62}
I20260620 23:04:08.657100 2739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:04:08.657549 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.657400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58514)
0620 23:04:08.657459 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.657516 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 23:04:08.657781 2689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
I20260620 23:04:08.658398 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.658276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58528)
0620 23:04:08.658334 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.658367 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 23:04:08.658545 2739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36571: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab.
Entry for principal kudu/127.21.66.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab.
I20260620 23:04:08.668716 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:36571
--webserver_interface=127.21.66.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab
--principal=kudu/127.21.66.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:38881,127.21.66.125:39937,127.21.66.124:36571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:08.707810 2739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } attempt: 1
I20260620 23:04:08.708566 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.708408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58540)
0620 23:04:08.708481 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.708530 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 23:04:08.709033 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.708916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58542)
0620 23:04:08.708971 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.709006 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:04:08.709172 2739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36571: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
I20260620 23:04:08.709350 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } attempt: 3
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:04:08.717207 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.709679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39336)
0620 23:04:08.709749 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.709762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.709822 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.709986 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.709987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.710238 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.710312 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.710314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.711301 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.711303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.711913 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.711915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.712437 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.712439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.712533 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.713028 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.713033 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.714421 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.716195 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.716196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.716197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.716406 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.716615 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.716615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.716616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.716673 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.716949 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.716950 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.716996 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.717056 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.717135 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 23:04:08.717257 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.709784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.21.66.125:39937)
0620 23:04:08.709843 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.709845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.709860 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:04:08.709889 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.709890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.709903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.709970 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.710389 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.711179 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.712009 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.712329 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.712402 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.714510 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.714512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.714512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.714519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.716104 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.716481 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.716482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.716483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.716541 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.716733 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.716733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.716734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.716884 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.716949 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.717095 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.717170 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:04:08.717826 2664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:04:08.718919 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.718752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58544)
0620 23:04:08.718844 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.718888 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:04:08.719278 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.719182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58548)
0620 23:04:08.719213 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.719248 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 23:04:08.719388 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36571: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
I20260620 23:04:08.750568 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } attempt: 1
I20260620 23:04:08.751165 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.750992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58560)
0620 23:04:08.751075 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.751125 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:04:08.751709 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.751601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58574)
0620 23:04:08.751652 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.751681 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:04:08.751884 2664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36571: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
W20260620 23:04:08.753104 2746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:08.753348 2746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:08.753374 2746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:08.754644 2746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:08.754684 2746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:08.754698 2746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:08.754711 2746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:08.754724 2746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:08.756217 2746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:38881,127.21.66.125:39937,127.21.66.124:36571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab
--principal=kudu/127.21.66.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.66.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:08.756418 2746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:08.756587 2746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:08.761909 2752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:08.761929 2754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:08.761909 2751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:08.762393 2746 server_base.cc:1034] running on GCE node
I20260620 23:04:08.762669 2746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:08.762979 2746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:08.764139 2746 hybrid_clock.cc:648] HybridClock initialized: now 1781996648764127 us; error 30 us; skew 500 ppm
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:08.767359 2746 init.cc:377] Logged in from keytab as kudu/127.21.66.124@KRBTEST.COM (short username kudu)
I20260620 23:04:08.767912 2746 webserver.cc:460] Webserver started at http://127.21.66.124:44245/ using document root <none> and password file <none>
I20260620 23:04:08.768190 2746 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:08.768270 2746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:08.768448 2746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:08.769613 2746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data/instance:
uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "5d7b0d8b2117a98baa0d3ed2deb23ab8"
server_key_iv: "a1b296eb344192eb24c7bfbd2e3a5835"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.770006 2746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/wal/instance:
uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "5d7b0d8b2117a98baa0d3ed2deb23ab8"
server_key_iv: "a1b296eb344192eb24c7bfbd2e3a5835"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.771445 2746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:08.772243 2761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:08.772468 2746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:08.772576 2746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/wal
uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3"
format_stamp: "Formatted at 2026-06-20 23:04:08 on dist-test-slave-qgtk"
server_key: "5d7b0d8b2117a98baa0d3ed2deb23ab8"
server_key_iv: "a1b296eb344192eb24c7bfbd2e3a5835"
server_key_version: "encryptionkey@0"
I20260620 23:04:08.772662 2746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:08.777473 2739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } attempt: 2
I20260620 23:04:08.777948 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.777810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58590)
0620 23:04:08.777878 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.777920 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 23:04:08.778370 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.778263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58596)
0620 23:04:08.778314 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.778341 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:04:08.778466 2739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36571: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36571: connect: Connection refused (error 111)
I20260620 23:04:08.797039 2746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:08.799647 2746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:08.799849 2746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:08.804123 2746 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.124:36571
I20260620 23:04:08.804159 2813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.124:36571 every 8 connection(s)
I20260620 23:04:08.804524 2746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
I20260620 23:04:08.805464 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:08.807116 2814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:08.814043 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2746
I20260620 23:04:08.814214 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781996487602640-21769-0/minicluster-data/master-2/wal/instance
I20260620 23:04:08.814466 21769 external_mini_cluster.cc:1428] Setting key 775127a10b3d83a1802714f8f4981092
I20260620 23:04:08.817400 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.807873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.21.66.126:38881)
0620 23:04:08.807935 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.807937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.808203 (+ 266us) server_negotiation.cc:407] Connection header received
0620 23:04:08.808577 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.808578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.808594 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.808662 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.809342 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.809868 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.810905 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.811109 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.811177 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.813647 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.813648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.813648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.813656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.815834 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.816350 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.816351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.816352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.816422 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.816668 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.816669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.816670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.816856 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.816938 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.817208 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.817297 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:04:08.817569 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.807822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:38881 (local address 127.0.0.1:52672)
0620 23:04:08.808001 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.808044 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.808480 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.808735 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.808738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.809014 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.809203 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.809206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.809956 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.809957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.810821 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.810823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.811196 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.811197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.811310 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.811847 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.811855 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.813526 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.815930 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.815937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.815940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.816273 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.816495 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.816496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.816496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.816599 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.816940 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.816945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.817065 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.817162 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.817315 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":20,"thread_start_us":103,"threads_started":1}
I20260620 23:04:08.818186 2814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:04:08.825585 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.814964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.21.66.124:36571)
0620 23:04:08.815186 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.815190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.815365 (+ 175us) server_negotiation.cc:407] Connection header received
0620 23:04:08.815399 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.815402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.815419 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.815514 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.816193 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.817452 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.818613 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.818916 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.818976 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.821801 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.821811 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.821812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.821819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.824004 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.824577 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.824581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.824581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.824643 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.824947 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.824948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.824948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.825153 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.825323 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.825375 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.825452 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260620 23:04:08.827032 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.818563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.21.66.125:39937)
0620 23:04:08.818617 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.818619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.819107 (+ 488us) server_negotiation.cc:407] Connection header received
0620 23:04:08.819145 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.819146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.819159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.819232 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.819568 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.820380 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.821412 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.821623 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.821724 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.823812 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.823814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.823814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.823823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.825761 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.826101 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.826102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.826103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.826185 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.826516 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.826518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.826518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.826639 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.826716 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.826866 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.826947 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:04:08.827030 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.818478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:39937 (local address 127.0.0.1:39346)
0620 23:04:08.819038 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.819046 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.819078 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.819247 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.819248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.819408 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.819462 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.819464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.820477 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.820478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.821270 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.821273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.821724 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.821725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.821809 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.822273 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.822278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.823709 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.825841 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.825842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.825842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.826025 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.826349 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.826350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.826350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.826407 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.826708 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.826708 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.826751 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.826840 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.826934 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548}
I20260620 23:04:08.827034 2664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } attempt: 2
I20260620 23:04:08.827652 2814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
I20260620 23:04:08.836163 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.827942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58634)
0620 23:04:08.827976 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.827985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.828013 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.828372 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.828373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.828530 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.828561 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.828562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.829478 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.829478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.830231 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.830234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.830997 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.830999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.831058 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.831483 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.831489 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.833039 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.835031 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.835032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.835032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.835262 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.835476 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.835477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.835477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.835530 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.835810 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.835811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.835850 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.835946 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.836044 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 23:04:08.836285 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.828039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.21.66.124:36571)
0620 23:04:08.828235 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.828237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.828244 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:04:08.828260 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.828261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.828270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.828357 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.828654 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.829394 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.830309 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.830616 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.830712 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.833141 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.833142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.833142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.833148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.834936 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.835345 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.835346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.835346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.835397 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.835602 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.835603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.835603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.835742 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.835818 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.835970 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.836108 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":178,"thread_start_us":127,"threads_started":1}
I20260620 23:04:08.836796 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.827264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58628)
0620 23:04:08.827316 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.827326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.827362 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.827937 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.827938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.828139 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.828188 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.828190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.829204 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.829205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.829911 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.829913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.830495 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.830496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.830567 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.830975 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.830979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.832488 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.835573 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.835574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.835574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.835792 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.836013 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.836014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.836014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.836097 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.836476 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.836477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.836518 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.836585 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.836681 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 23:04:08.837180 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.826528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.21.66.126:38881)
0620 23:04:08.826572 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.826574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.826581 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:04:08.826609 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.826610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.826631 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.826695 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.827262 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.827951 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.828881 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.829175 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.829215 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.833697 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.833699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.833699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.833708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.835571 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.836025 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.836026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.836027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.836109 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.836353 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.836354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.836354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.836496 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.836685 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.837006 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.837082 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:04:08.837489 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.827370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.21.66.124:36571)
0620 23:04:08.827825 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.827827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.827835 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:04:08.827852 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.827853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.827862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.827927 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.828386 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.829113 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.830028 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.830411 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.830483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.832582 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.832584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.832584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.832591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.835470 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.835879 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.835880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.835880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.835938 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.836176 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.836177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.836178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.836384 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.836448 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.837267 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.837373 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439}
I20260620 23:04:08.838322 2814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3: Bootstrap starting.
I20260620 23:04:08.839079 2814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:08.839375 2814 log.cc:824] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:08.840152 2814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3: No bootstrap required, opened a new log
I20260620 23:04:08.841240 2664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: Bootstrap starting.
I20260620 23:04:08.842051 2664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:08.841993 2814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:08.842170 2814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:08.842232 2814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d02a351eabf4d3ca4f7fe0fe77dfcc3, State: Initialized, Role: FOLLOWER
I20260620 23:04:08.842365 2814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:08.842449 2664 log.cc:824] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:08.842691 2814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:08.842903 2821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:08.843096 2821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:04:08.843307 2664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: No bootstrap required, opened a new log
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
W20260620 23:04:08.844587 2833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:04:08.844695 2833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:04:08.846433 2664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:08.846576 2664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:08.846613 2664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aaaef84f99b4a29aa85756c5deef09f, State: Initialized, Role: FOLLOWER
I20260620 23:04:08.846722 2664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:08.847164 2664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:08.845010 2746 master_runner.cc:186] Error getting master registration for 127.21.66.126:38881: OK, instance_id { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" instance_seqno: 1781996648480224 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:04:08.847580 2746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:04:08.847945 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.837824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.21.66.125:39937)
0620 23:04:08.837871 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.837873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.837881 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:04:08.837905 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.837906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.837918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.837991 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.838642 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.839339 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.840391 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.840666 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.840744 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.843857 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.843858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.843859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.843868 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.846060 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.846671 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.846672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.846673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.846737 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.846986 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.846987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.846987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.847146 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.847338 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.847744 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.847844 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:04:08.848150 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:08.848232 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:04:08.849001 2846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:04:08.850003 2846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:04:08.850600 2596 master_runner.cc:186] Error getting master registration for 127.21.66.125:39937: OK, instance_id { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" instance_seqno: 1781996648631948 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:04:08.852639 2596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:04:08.852869 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:04:08.863235 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.855027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.21.66.124:36571)
0620 23:04:08.855081 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.855083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.855091 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:04:08.855116 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.855117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.855127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.855179 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.855960 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.856735 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.857875 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.858208 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.858245 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.860182 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.860183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.860183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.860190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.861887 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.862323 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.862324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.862324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.862383 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.862596 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.862597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.862597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.862736 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.862811 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.863047 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.863140 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:04:08.864378 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.855255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.21.66.126:38881)
0620 23:04:08.855294 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.855296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.855303 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:04:08.855324 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.855325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.855345 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.855396 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.856037 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.856734 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.857866 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.858195 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.858241 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.860251 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.860253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.860253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.860259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.862504 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.862856 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.862857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.862857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.862913 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.863125 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.863126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.863126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.863244 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.863747 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.864190 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.864263 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:04:08.869618 2739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } attempt: 3
I20260620 23:04:08.871025 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.855074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.21.66.125:39937)
0620 23:04:08.855113 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.855115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.855131 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:04:08.855149 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.855150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.855159 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.855206 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.865118 (+ 9912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.865586 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.866369 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.866571 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.866646 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.867871 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.867873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.867873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.867881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.869705 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.870108 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.870109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.870109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.870203 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.870430 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.870431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.870431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.870563 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.870640 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.870856 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.870933 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 23:04:08 dist-test-slave-qgtk krb5kdc[2583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996648, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
I20260620 23:04:08.877223 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.869873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36571 (local address 127.0.0.1:58642)
0620 23:04:08.869965 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:08.869976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:08.870012 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:08.870375 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:08.870376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:08.870529 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:08.870570 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.870572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.871433 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.871434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.872156 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:08.872158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.872520 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.872521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:08.872577 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.873000 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:08.873005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:08.874216 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:08.876096 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.876097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.876098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.876364 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.876558 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:08.876559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:08.876560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:08.876615 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:08.876903 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:08.876904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:08.876945 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:04:08.877029 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:04:08.877127 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:04:08.877221 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:08.870088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.21.66.124:36571)
0620 23:04:08.870236 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:08.870238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:08.870246 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:04:08.870268 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:08.870269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:08.870281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:08.870329 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:08.870648 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.871348 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.872244 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:08.872444 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:08.872519 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:08.874313 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:08.874314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:08.874315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:08.874321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:08.876019 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.876443 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.876444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.876444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.876502 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:08.876681 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:08.876682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:08.876683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:08.876806 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:08.876886 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:08.877054 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 23:04:08.877127 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 23:04:08.878703 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: Bootstrap starting.
I20260620 23:04:08.879423 2739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:08.879733 2739 log.cc:824] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:08.880527 2739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: No bootstrap required, opened a new log
I20260620 23:04:08.881953 2739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:08.882051 2739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:08.882071 2739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9493b2dff88e4e93b6af94e0a81a750c, State: Initialized, Role: FOLLOWER
I20260620 23:04:08.882269 2739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:08.883059 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:08.883126 2739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:08.883177 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:04:08.884975 2867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:04:08.885057 2867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:04:08.887075 2671 master_runner.cc:263] No leader master found from master 9493b2dff88e4e93b6af94e0a81a750c
I20260620 23:04:08.887120 2671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:04:08.977815 2617 tablet.cc:2369] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:04:09.050534 2767 tablet.cc:2369] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:04:09.057446 2821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:09.057577 2821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:09.057948 2821 leader_election.cc:288] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4aaaef84f99b4a29aa85756c5deef09f (127.21.66.126:38881), 9493b2dff88e4e93b6af94e0a81a750c (127.21.66.125:39937)
I20260620 23:04:09.058399 2713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9493b2dff88e4e93b6af94e0a81a750c" is_pre_election: true
I20260620 23:04:09.058408 2638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aaaef84f99b4a29aa85756c5deef09f" is_pre_election: true
I20260620 23:04:09.058576 2713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d02a351eabf4d3ca4f7fe0fe77dfcc3 in term 0.
I20260620 23:04:09.058578 2638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d02a351eabf4d3ca4f7fe0fe77dfcc3 in term 0.
I20260620 23:04:09.058791 2764 leader_election.cc:302] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aaaef84f99b4a29aa85756c5deef09f, 7d02a351eabf4d3ca4f7fe0fe77dfcc3; no voters:
I20260620 23:04:09.058981 2821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:04:09.059054 2821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:04:09.059074 2821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:09.059831 2821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:09.059973 2821 leader_election.cc:288] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [CANDIDATE]: Term 1 election: Requested vote from peers 4aaaef84f99b4a29aa85756c5deef09f (127.21.66.126:38881), 9493b2dff88e4e93b6af94e0a81a750c (127.21.66.125:39937)
I20260620 23:04:09.060146 2713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9493b2dff88e4e93b6af94e0a81a750c"
I20260620 23:04:09.060148 2638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aaaef84f99b4a29aa85756c5deef09f"
I20260620 23:04:09.060240 2713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:09.060238 2638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:09.061105 2638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d02a351eabf4d3ca4f7fe0fe77dfcc3 in term 1.
I20260620 23:04:09.061106 2713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d02a351eabf4d3ca4f7fe0fe77dfcc3 in term 1.
I20260620 23:04:09.061339 2764 leader_election.cc:302] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4aaaef84f99b4a29aa85756c5deef09f, 7d02a351eabf4d3ca4f7fe0fe77dfcc3; no voters:
I20260620 23:04:09.061443 2821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:09.061681 2821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7d02a351eabf4d3ca4f7fe0fe77dfcc3, State: Running, Role: LEADER
I20260620 23:04:09.061826 2821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } }
I20260620 23:04:09.062341 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d02a351eabf4d3ca4f7fe0fe77dfcc3. Latest consensus state: current_term: 1 leader_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:09.062427 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:09.062652 2871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:09.062891 2871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:09.064414 2713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [term 1 FOLLOWER]: Refusing update from remote peer 7d02a351eabf4d3ca4f7fe0fe77dfcc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:09.064457 2638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [term 1 FOLLOWER]: Refusing update from remote peer 7d02a351eabf4d3ca4f7fe0fe77dfcc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:09.064777 2821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:09.064924 2869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:09.066213 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d02a351eabf4d3ca4f7fe0fe77dfcc3. Latest consensus state: current_term: 1 leader_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:09.066294 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:04:09.066386 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d02a351eabf4d3ca4f7fe0fe77dfcc3. Latest consensus state: current_term: 1 leader_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:09.066496 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:04:09.067885 2821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:09.067991 2821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:09.068240 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:09.068343 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:04:09.068461 2871 catalog_manager.cc:1347] Generated new cluster ID: b6e2745e2c7a4c4db1d91130aea85bce
I20260620 23:04:09.068512 2871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:09.068492 2873 mvcc.cc:205] Tried to move back new op lower bound from 7299058274565300224 to 7299058274558005248. Current Snapshot: MvccSnapshot[applied={T|T < 7299058274565300224}]
I20260620 23:04:09.068477 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:09.068552 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:04:09.067898 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4aaaef84f99b4a29aa85756c5deef09f" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 38881 } } peers { permanent_uuid: "9493b2dff88e4e93b6af94e0a81a750c" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 39937 } } peers { permanent_uuid: "7d02a351eabf4d3ca4f7fe0fe77dfcc3" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36571 } } }
I20260620 23:04:09.068758 2874 mvcc.cc:205] Tried to move back new op lower bound from 7299058274565300224 to 7299058274558005248. Current Snapshot: MvccSnapshot[applied={T|T < 7299058274565300224}]
I20260620 23:04:09.068668 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:09.085035 2871 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:04:09.085716 2871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:09.089203 2871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d02a351eabf4d3ca4f7fe0fe77dfcc3: Generated new TSK 0
I20260620 23:04:09.089526 2871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:04:09.850528 2846 catalog_manager.cc:1259] Loaded cluster ID: b6e2745e2c7a4c4db1d91130aea85bce
I20260620 23:04:09.850618 2846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: loading cluster ID for follower catalog manager: success
I20260620 23:04:09.851490 2846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: acquiring CA information for follower catalog manager: success
I20260620 23:04:09.851809 2846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4aaaef84f99b4a29aa85756c5deef09f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:04:09.885591 2867 catalog_manager.cc:1259] Loaded cluster ID: b6e2745e2c7a4c4db1d91130aea85bce
I20260620 23:04:09.885668 2867 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: loading cluster ID for follower catalog manager: success
I20260620 23:04:09.886473 2867 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: acquiring CA information for follower catalog manager: success
I20260620 23:04:09.886724 2867 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9493b2dff88e4e93b6af94e0a81a750c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:04:10.155112 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2596
I20260620 23:04:10.160944 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2671
I20260620 23:04:10.165994 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2746
2026-06-20T23:04:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3040 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:04:10 dist-test-slave-qgtk krb5kdc[2886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:04:10 dist-test-slave-qgtk krb5kdc[2886](info): set up 2 sockets
Jun 20 23:04:10 dist-test-slave-qgtk krb5kdc[2886](info): commencing operation
krb5kdc: starting...
W20260620 23:04:11.318847 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:04:11 dist-test-slave-qgtk krb5kdc[2886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:04:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:04:11.450556 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:35569
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:11.532807 2899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:11.533073 2899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:11.533111 2899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:11.534358 2899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:11.534435 2899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:11.534459 2899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:11.534480 2899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:11.534508 2899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:11.536036 2899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:35569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:11.536321 2899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:11.536554 2899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:11.541812 2904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:11.541828 2905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:11.541828 2907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:11.542182 2899 server_base.cc:1034] running on GCE node
I20260620 23:04:11.542475 2899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:11.542850 2899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:11.544068 2899 hybrid_clock.cc:648] HybridClock initialized: now 1781996651544039 us; error 45 us; skew 500 ppm
Jun 20 23:04:11 dist-test-slave-qgtk krb5kdc[2886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996651, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:11.547019 2899 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:11.547498 2899 webserver.cc:460] Webserver started at http://127.21.66.126:34261/ using document root <none> and password file <none>
I20260620 23:04:11.547734 2899 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:11.547804 2899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:11.547959 2899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:11.549043 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "e78381f6e0b8443e9f5df5805bec7bc6"
format_stamp: "Formatted at 2026-06-20 23:04:11 on dist-test-slave-qgtk"
server_key: "360894190b1662f46fa63d92470c0b95"
server_key_iv: "654476040691c923a5b357ee064c41a0"
server_key_version: "encryptionkey@0"
I20260620 23:04:11.549413 2899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "e78381f6e0b8443e9f5df5805bec7bc6"
format_stamp: "Formatted at 2026-06-20 23:04:11 on dist-test-slave-qgtk"
server_key: "360894190b1662f46fa63d92470c0b95"
server_key_iv: "654476040691c923a5b357ee064c41a0"
server_key_version: "encryptionkey@0"
I20260620 23:04:11.550928 2899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:04:11.551774 2914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:11.551975 2899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:11.552083 2899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "e78381f6e0b8443e9f5df5805bec7bc6"
format_stamp: "Formatted at 2026-06-20 23:04:11 on dist-test-slave-qgtk"
server_key: "360894190b1662f46fa63d92470c0b95"
server_key_iv: "654476040691c923a5b357ee064c41a0"
server_key_version: "encryptionkey@0"
I20260620 23:04:11.552160 2899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:11.582871 2899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:11.585359 2899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:11.585546 2899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:11.589800 2899 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:35569
I20260620 23:04:11.589838 2966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:35569 every 8 connection(s)
I20260620 23:04:11.590220 2899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:11.591151 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:11.593644 2967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6: Bootstrap starting.
I20260620 23:04:11.594305 2967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:11.594635 2967 log.cc:824] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:11.595414 2967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6: No bootstrap required, opened a new log
I20260620 23:04:11.596310 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 2899
I20260620 23:04:11.596447 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:04:11.596638 21769 external_mini_cluster.cc:1428] Setting key 1c22be33213c48de458c17b86d2621bf
I20260620 23:04:11.597595 2967 raft_consensus.cc:348] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35569 } }
I20260620 23:04:11.597786 2967 raft_consensus.cc:374] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:11.597831 2967 raft_consensus.cc:729] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78381f6e0b8443e9f5df5805bec7bc6, State: Initialized, Role: FOLLOWER
I20260620 23:04:11.597970 2967 consensus_queue.cc:260] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35569 } }
I20260620 23:04:11.598055 2967 raft_consensus.cc:388] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:11.598109 2967 raft_consensus.cc:482] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:11.598187 2967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:11.599228 2967 raft_consensus.cc:504] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35569 } }
I20260620 23:04:11.599380 2967 leader_election.cc:302] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e78381f6e0b8443e9f5df5805bec7bc6; no voters:
I20260620 23:04:11.599529 2967 leader_election.cc:288] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:04:11.599597 2972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:11.599781 2972 raft_consensus.cc:686] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [term 1 LEADER]: Becoming Leader. State: Replica: e78381f6e0b8443e9f5df5805bec7bc6, State: Running, Role: LEADER
I20260620 23:04:11.599931 2967 sys_catalog.cc:564] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:11.599900 2972 consensus_queue.cc:237] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35569 } }
I20260620 23:04:11.600363 2972 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35569 } } }
I20260620 23:04:11.600453 2972 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:11.600610 2972 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e78381f6e0b8443e9f5df5805bec7bc6. Latest consensus state: current_term: 1 leader_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e78381f6e0b8443e9f5df5805bec7bc6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35569 } } }
I20260620 23:04:11.600677 2972 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6 [sys.catalog]: This master's current role is: LEADER
W20260620 23:04:11.602301 2988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:04:11.602370 2988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:04:11.602424 2977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:11.602609 2977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:04:11 dist-test-slave-qgtk krb5kdc[2886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:11.604796 2977 catalog_manager.cc:1347] Generated new cluster ID: a3e4bb116b544ea3ae63aa702b28ebfa
I20260620 23:04:11.604861 2977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:11.609241 2971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:11.597249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.21.66.126:35569)
0620 23:04:11.597480 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:11.597490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:11.597531 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:04:11.597652 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:11.597659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:11.597716 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:11.597889 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:11.598803 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:11.600231 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:11.601436 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:11.602098 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:11.602230 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:11.605260 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:11.605265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:11.605265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:11.605278 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:11.607459 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:11.607898 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:11.607901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:11.607902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:11.608029 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:11.608285 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:11.608286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:11.608286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:11.608596 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:11.608761 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:11.608871 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:04:11.609012 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260620 23:04:11.610383 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:04:11.622952 2977 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:04:11.623883 2977 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:11.628412 2977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e78381f6e0b8443e9f5df5805bec7bc6: Generated new TSK 0
I20260620 23:04:11.628636 2977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:04:11.664608 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 2899
2026-06-20T23:04:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1496 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:04:11 dist-test-slave-qgtk krb5kdc[2994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:04:11 dist-test-slave-qgtk krb5kdc[2994](info): set up 2 sockets
Jun 20 23:04:11 dist-test-slave-qgtk krb5kdc[2994](info): commencing operation
krb5kdc: starting...
W20260620 23:04:12.714378 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:04:12 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:04:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:04:12.841747 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40885
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:12.934851 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:12.935061 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:12.935083 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:12.936339 3007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:12.936386 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:12.936400 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:12.936414 3007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:12.936425 3007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:12.937917 3007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:12.938200 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:12.938441 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:12.943503 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:12.943519 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:12.943529 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:12.943950 3007 server_base.cc:1034] running on GCE node
I20260620 23:04:12.944247 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:12.944557 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:12.945756 3007 hybrid_clock.cc:648] HybridClock initialized: now 1781996652945749 us; error 41 us; skew 500 ppm
Jun 20 23:04:12 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996652, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:12.948519 3007 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:12.948967 3007 webserver.cc:460] Webserver started at http://127.21.66.126:46771/ using document root <none> and password file <none>
I20260620 23:04:12.949179 3007 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:12.949221 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:12.949312 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:12.950444 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "254a17821b47481ab4cf54c9e72e8938"
format_stamp: "Formatted at 2026-06-20 23:04:12 on dist-test-slave-qgtk"
server_key: "5c8fd9afb1bb8ddf037683acda8d6cd0"
server_key_iv: "a6e77f7338aa28ec5332f2472223f92e"
server_key_version: "encryptionkey@0"
I20260620 23:04:12.950766 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "254a17821b47481ab4cf54c9e72e8938"
format_stamp: "Formatted at 2026-06-20 23:04:12 on dist-test-slave-qgtk"
server_key: "5c8fd9afb1bb8ddf037683acda8d6cd0"
server_key_iv: "a6e77f7338aa28ec5332f2472223f92e"
server_key_version: "encryptionkey@0"
I20260620 23:04:12.952082 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:12.952893 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:12.953171 3007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:12.953260 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "254a17821b47481ab4cf54c9e72e8938"
format_stamp: "Formatted at 2026-06-20 23:04:12 on dist-test-slave-qgtk"
server_key: "5c8fd9afb1bb8ddf037683acda8d6cd0"
server_key_iv: "a6e77f7338aa28ec5332f2472223f92e"
server_key_version: "encryptionkey@0"
I20260620 23:04:12.953325 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:12.977058 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:12.979439 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:12.979560 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:12.983630 3007 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:40885
I20260620 23:04:12.983692 3074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:40885 every 8 connection(s)
I20260620 23:04:12.984017 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:12.985109 3075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:12.986297 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3007
I20260620 23:04:12.986426 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:04:12.986624 21769 external_mini_cluster.cc:1428] Setting key 76a5f3859b91a7f5295ca986f0a746fa
I20260620 23:04:12.988705 3075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Bootstrap starting.
I20260620 23:04:12.989316 3075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:12.989626 3075 log.cc:824] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:12.990376 3075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: No bootstrap required, opened a new log
I20260620 23:04:12.992379 3075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:12.992532 3075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:12.992615 3075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a17821b47481ab4cf54c9e72e8938, State: Initialized, Role: FOLLOWER
I20260620 23:04:12.992795 3075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:12.992897 3075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:12.992931 3075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:12.992988 3075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:04:12 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:12.993731 3075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:12.993862 3075 leader_election.cc:302] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254a17821b47481ab4cf54c9e72e8938; no voters:
I20260620 23:04:12.994035 3075 leader_election.cc:288] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:04:12.994092 3080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:12.994328 3075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:12.994364 3080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 LEADER]: Becoming Leader. State: Replica: 254a17821b47481ab4cf54c9e72e8938, State: Running, Role: LEADER
I20260620 23:04:12.994511 3080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:12.994886 3081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "254a17821b47481ab4cf54c9e72e8938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } } }
I20260620 23:04:12.994997 3081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:12.994886 3082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254a17821b47481ab4cf54c9e72e8938. Latest consensus state: current_term: 1 leader_uuid: "254a17821b47481ab4cf54c9e72e8938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } } }
I20260620 23:04:12.995205 3082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:12.995795 3089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:12.996466 3089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:12.998055 3089 catalog_manager.cc:1347] Generated new cluster ID: b85089f78dfd4f998a5118723defacb8
I20260620 23:04:12.998135 3089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:12.998847 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:12.987246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.21.66.126:40885)
0620 23:04:12.987442 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:12.987452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:12.987489 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:04:12.987931 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:12.987938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:12.987986 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:12.988152 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:12.988783 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:12.989818 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:12.990748 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:12.991172 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:12.991299 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:12.994077 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:12.994082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:12.994083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:12.994097 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:12.996919 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:12.997508 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:12.997512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:12.997512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:12.997627 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:12.997876 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:12.997877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:12.997877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:12.998212 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:12.998362 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:12.998487 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 23:04:12.998636 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260620 23:04:13.012382 3089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:04:13.012992 3089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:13.019448 3089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Generated new TSK 0
I20260620 23:04:13.019682 3089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:04:13.056617 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40885
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:13.140836 3101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:13.141040 3101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:13.141076 3101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:13.142334 3101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:13.142376 3101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:13.142403 3101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:04:13.143896 3101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:13.144155 3101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:13.144407 3101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:13.149791 3107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:13.149806 3109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:13.149976 3101 server_base.cc:1034] running on GCE node
W20260620 23:04:13.149787 3106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:13.150384 3101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:13.150681 3101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:13.151950 3101 hybrid_clock.cc:648] HybridClock initialized: now 1781996653151927 us; error 39 us; skew 500 ppm
Jun 20 23:04:13 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:13.154912 3101 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:04:13.155494 3101 webserver.cc:460] Webserver started at http://127.21.66.65:35515/ using document root <none> and password file <none>
I20260620 23:04:13.155750 3101 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:13.155827 3101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:13.155989 3101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:13.157231 3101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "fc65870f1c874e969af2065851e597de"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "faf4153f449cd1f3d4b58d654893a018"
server_key_iv: "33bcb0f28a4f6855312a210c375493ef"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.157697 3101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "fc65870f1c874e969af2065851e597de"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "faf4153f449cd1f3d4b58d654893a018"
server_key_iv: "33bcb0f28a4f6855312a210c375493ef"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.159344 3101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:04:13.160351 3116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.160567 3101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:13.160692 3101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "fc65870f1c874e969af2065851e597de"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "faf4153f449cd1f3d4b58d654893a018"
server_key_iv: "33bcb0f28a4f6855312a210c375493ef"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.160780 3101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:13.180650 3101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:13.183297 3101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:13.183497 3101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:13.183806 3101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:13.184165 3101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:13.184226 3101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.184265 3101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:13.184369 3101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.190781 3101 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:43995
I20260620 23:04:13.190959 3229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:43995 every 8 connection(s)
I20260620 23:04:13.191236 3101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:13.192278 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3101
I20260620 23:04:13.192417 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:04:13.192690 21769 external_mini_cluster.cc:1428] Setting key d0de3f156eb6fbd9fe9fa74f62b98a32
Jun 20 23:04:13 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:04:13.203658 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.192133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:39503 (local address 127.21.66.126:40885)
0620 23:04:13.192192 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.192195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.192509 (+ 314us) server_negotiation.cc:407] Connection header received
0620 23:04:13.193280 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.193282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.193294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.193363 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:13.195043 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.196096 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.197281 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.197493 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.197608 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.200216 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:13.200218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:13.200218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:13.200226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:13.201861 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.202453 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.202455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.202455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.202523 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.202835 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.202836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.202836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.202972 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:13.203054 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.203451 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.203552 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 23:04:13.203874 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.192024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:39503)
0620 23:04:13.192346 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.192409 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.193169 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.193485 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.193490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.194603 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:13.194934 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.194942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.196215 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.196216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.197157 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.197160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.197619 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.197620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.197755 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.198439 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:13.198448 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:13.200061 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:13.202066 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:13.202075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:13.202077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:13.202346 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:13.202601 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:13.202602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:13.202603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:13.202749 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:13.203060 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:13.203068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:13.203256 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.203414 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.203595 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
I20260620 23:04:13.204530 3230 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40885
I20260620 23:04:13.204620 3230 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
I20260620 23:04:13.204859 3230 heartbeater.cc:511] Master 127.21.66.126:40885 requested a full tablet report, sending...
I20260620 23:04:13.205741 3039 ts_manager.cc:194] Registered new tserver with Master: fc65870f1c874e969af2065851e597de (127.21.66.65:43995)
I20260620 23:04:13.206492 3039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:39503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:04:13.222376 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40885
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:13.308543 3235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:13.308817 3235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:13.308859 3235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:13.310071 3235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:13.310175 3235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:13.310235 3235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:04:13.311748 3235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:13.312036 3235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:13.312299 3235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:13.317520 3240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:13.317646 3241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:13.318264 3243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:13.318403 3235 server_base.cc:1034] running on GCE node
I20260620 23:04:13.318639 3235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:13.318912 3235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:13.320072 3235 hybrid_clock.cc:648] HybridClock initialized: now 1781996653320059 us; error 35 us; skew 500 ppm
Jun 20 23:04:13 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:13.323721 3235 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:04:13.324308 3235 webserver.cc:460] Webserver started at http://127.21.66.66:45463/ using document root <none> and password file <none>
I20260620 23:04:13.324549 3235 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:13.324617 3235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:13.324751 3235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:13.325867 3235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "348cea95e6314169be2b18059462e979"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "c1ecd62e3eb09e626aaa78875ab0bbb4"
server_key_iv: "5bc159ff091b959922d56c106fe71819"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.326434 3235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "348cea95e6314169be2b18059462e979"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "c1ecd62e3eb09e626aaa78875ab0bbb4"
server_key_iv: "5bc159ff091b959922d56c106fe71819"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.327841 3235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:13.328608 3250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.328835 3235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:13.328953 3235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "348cea95e6314169be2b18059462e979"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "c1ecd62e3eb09e626aaa78875ab0bbb4"
server_key_iv: "5bc159ff091b959922d56c106fe71819"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.329049 3235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:13.373248 3235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:13.375986 3235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:13.376206 3235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:13.376482 3235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:13.376832 3235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:13.376894 3235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.376942 3235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:13.376973 3235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.383881 3235 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:39931
I20260620 23:04:13.383936 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:39931 every 8 connection(s)
I20260620 23:04:13.384282 3235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:04:13.389693 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3235
I20260620 23:04:13.389915 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:04:13.390247 21769 external_mini_cluster.cc:1428] Setting key ebc6fc04149ab448408052ad709a919e
Jun 20 23:04:13 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:13.395521 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.385072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:39577 (local address 127.21.66.126:40885)
0620 23:04:13.385119 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.385121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.385364 (+ 243us) server_negotiation.cc:407] Connection header received
0620 23:04:13.385908 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.385909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.385921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.385986 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:13.387004 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.387849 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.389000 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.389198 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.389259 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.391864 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:13.391865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:13.391865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:13.391872 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:13.393820 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.394403 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.394404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.394404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.394471 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.394771 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.394772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.394773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.394903 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:13.394984 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.395316 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.395407 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:04:13.395846 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.385034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:39577)
0620 23:04:13.385218 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.385265 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.385811 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.386045 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.386048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.386697 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:13.386911 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.386916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.387961 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.387962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.388908 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.388910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.389284 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.389285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.389397 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.390076 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:13.390087 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:13.391764 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:13.393906 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:13.393912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:13.393914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:13.394314 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:13.394541 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:13.394542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:13.394542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:13.394691 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:13.394973 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:13.394980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:13.395175 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.395283 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.395454 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260620 23:04:13.396318 3364 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40885
I20260620 23:04:13.396442 3364 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:13.396677 3364 heartbeater.cc:511] Master 127.21.66.126:40885 requested a full tablet report, sending...
I20260620 23:04:13.397109 3039 ts_manager.cc:194] Registered new tserver with Master: 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
I20260620 23:04:13.397511 3039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:39577
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:04:13.418959 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40885
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:13.504513 3369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:13.504756 3369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:13.504797 3369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:13.505944 3369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:13.506009 3369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:13.506054 3369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:04:13.507519 3369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:13.507795 3369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:13.508024 3369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:13.513556 3377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:13.513556 3375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:13.513799 3374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:13.513834 3369 server_base.cc:1034] running on GCE node
I20260620 23:04:13.514147 3369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:13.514436 3369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:13.515605 3369 hybrid_clock.cc:648] HybridClock initialized: now 1781996653515591 us; error 27 us; skew 500 ppm
Jun 20 23:04:13 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:13.521742 3369 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:04:13.522225 3369 webserver.cc:460] Webserver started at http://127.21.66.67:33297/ using document root <none> and password file <none>
I20260620 23:04:13.522436 3369 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:13.522480 3369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:13.522572 3369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:13.523623 3369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "05c5b3073fcadf456e616d188329f6ed"
server_key_iv: "c5c2265f58cd538189377c000973f557"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.523942 3369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "05c5b3073fcadf456e616d188329f6ed"
server_key_iv: "c5c2265f58cd538189377c000973f557"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.525305 3369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:04:13.526055 3384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.526347 3369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:13.526450 3369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "05c5b3073fcadf456e616d188329f6ed"
server_key_iv: "c5c2265f58cd538189377c000973f557"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.526544 3369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:13.538028 3369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:13.540398 3369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:13.540560 3369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:13.540802 3369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:13.541116 3369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:13.541149 3369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.541172 3369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:13.541219 3369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.547032 3369 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:33503
I20260620 23:04:13.547114 3497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:33503 every 8 connection(s)
I20260620 23:04:13.547430 3369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:04:13 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:13.554371 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3369
I20260620 23:04:13.554512 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:04:13.554756 21769 external_mini_cluster.cc:1428] Setting key 2fef992d15e0f56f444b4732a903dcc7
I20260620 23:04:13.558017 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.548278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:38167 (local address 127.21.66.126:40885)
0620 23:04:13.548345 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.548347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.548540 (+ 193us) server_negotiation.cc:407] Connection header received
0620 23:04:13.549064 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.549065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.549075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.549196 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:13.550350 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.550893 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.551978 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.552230 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.552290 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.554684 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:13.554686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:13.554686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:13.554695 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:13.556364 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.556937 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.556938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.556939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.556996 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.557362 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.557363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.557364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.557496 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:13.557566 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.557870 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.557948 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:04:13.558269 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.548243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:38167)
0620 23:04:13.548410 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.548455 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.548971 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.549289 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.549293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.549939 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:13.550222 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.550229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.550987 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.550988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.551893 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.551895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.552312 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.552312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.552420 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.552912 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:13.552921 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:13.554556 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:13.556444 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:13.556451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:13.556453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:13.556820 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:13.557068 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:13.557068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:13.557069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:13.557271 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:13.557566 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:13.557571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:13.557702 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.557810 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.557947 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260620 23:04:13.558903 3498 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40885
I20260620 23:04:13.559015 3498 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:13.559264 3498 heartbeater.cc:511] Master 127.21.66.126:40885 requested a full tablet report, sending...
I20260620 23:04:13.559705 3039 ts_manager.cc:194] Registered new tserver with Master: 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503)
I20260620 23:04:13.560107 3039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:38167
I20260620 23:04:13.566716 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:04:13.574050 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.568175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.21.66.126:40885)
0620 23:04:13.568228 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.568229 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.568237 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:04:13.568275 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.568276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.568284 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.568335 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:13.568904 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.569447 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.570229 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.570423 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.570480 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.571688 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:13.571690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:13.571690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:13.571696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:13.572956 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.573309 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.573310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.573310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.573364 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:13.573531 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:13.573531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:13.573531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:13.573681 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:13.573759 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.573917 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.573983 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:04:13.575932 3039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:04:13.577497 3039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:04:13.584090 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.579859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43995 (local address 127.0.0.1:37934)
0620 23:04:13.580282 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.580293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.580342 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.580611 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.580611 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.580614 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:13.580660 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.580662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.581985 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.581986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.583073 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.583075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.583755 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.583756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.583835 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.583856 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.583944 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.584011 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":67,"thread_start_us":57,"threads_started":1}
I20260620 23:04:13.584106 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.580049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.21.66.65:43995)
0620 23:04:13.580302 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.580306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.580379 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:04:13.580423 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.580426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.580443 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.580515 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:13.580718 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.581877 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.583166 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.583655 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.583751 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.583810 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.583962 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.584007 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":136,"threads_started":1}
I20260620 23:04:13.584199 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.579761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.21.66.66:39931)
0620 23:04:13.580079 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.580084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.580093 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:13.580134 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.580136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.580148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.580255 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:13.580471 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.581462 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.582878 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.583657 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.583725 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.583786 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.584055 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.584102 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":87,"threads_started":1}
I20260620 23:04:13.584259 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.579671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:39931 (local address 127.0.0.1:51966)
0620 23:04:13.579842 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.579885 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.579952 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.580291 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.580295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.580297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:13.580372 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.580374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.581565 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.581566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.582780 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.582782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.583819 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.583820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.583883 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.583887 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.583932 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.584010 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":45,"threads_started":1}
I20260620 23:04:13.584595 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.579702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33503 (local address 127.0.0.1:48792)
0620 23:04:13.579932 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.579944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.579996 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.580698 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.580699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.580701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:13.580757 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.580758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.582026 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.582027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.583361 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.583364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.584332 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.584333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.584408 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.584413 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.584467 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.584501 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260620 23:04:13.584869 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.579855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.21.66.67:33503)
0620 23:04:13.580468 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.580473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.580482 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:13.580521 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.580522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.580538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.580612 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:13.580843 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.581920 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.583462 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.584179 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.584247 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.584299 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.584729 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.584774 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":85,"threads_started":1}
I20260620 23:04:13.585366 3298 tablet_service.cc:1467] Processing CreateTablet for tablet 50ee32751b1e4a1ea8c89d14565a3155 (DEFAULT_TABLE table=test-table [id=a6506aa83b99439e98c2b9edc422b91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:13.585466 3432 tablet_service.cc:1467] Processing CreateTablet for tablet 50ee32751b1e4a1ea8c89d14565a3155 (DEFAULT_TABLE table=test-table [id=a6506aa83b99439e98c2b9edc422b91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:13.585753 3432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ee32751b1e4a1ea8c89d14565a3155. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:13.585753 3298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ee32751b1e4a1ea8c89d14565a3155. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:13.589183 3516 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Bootstrap starting.
I20260620 23:04:13.589933 3516 tablet_bootstrap.cc:654] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:13.590291 3164 tablet_service.cc:1467] Processing CreateTablet for tablet 50ee32751b1e4a1ea8c89d14565a3155 (DEFAULT_TABLE table=test-table [id=a6506aa83b99439e98c2b9edc422b91f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:13.590628 3164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ee32751b1e4a1ea8c89d14565a3155. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:13.590727 3516 log.cc:824] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:13.591066 3515 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Bootstrap starting.
I20260620 23:04:13.591506 3516 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: No bootstrap required, opened a new log
I20260620 23:04:13.591578 3516 ts_tablet_manager.cc:1397] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:13.591753 3515 tablet_bootstrap.cc:654] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:13.592244 3515 log.cc:824] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:13.592931 3515 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: No bootstrap required, opened a new log
I20260620 23:04:13.593014 3515 ts_tablet_manager.cc:1397] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:04:13.593585 3516 raft_consensus.cc:348] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.593772 3516 raft_consensus.cc:374] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:13.593822 3516 raft_consensus.cc:729] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348cea95e6314169be2b18059462e979, State: Initialized, Role: FOLLOWER
I20260620 23:04:13.593981 3519 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Bootstrap starting.
I20260620 23:04:13.593962 3516 consensus_queue.cc:260] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.594286 3516 ts_tablet_manager.cc:1428] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:04:13.594437 3364 heartbeater.cc:503] Master 127.21.66.126:40885 was elected leader, sending a full tablet report...
I20260620 23:04:13.594738 3519 tablet_bootstrap.cc:654] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:13.594909 3515 raft_consensus.cc:348] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.595088 3515 raft_consensus.cc:374] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:13.595124 3515 raft_consensus.cc:729] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7019f5cded4548dfbafcb2d5a2ca6ccd, State: Initialized, Role: FOLLOWER
I20260620 23:04:13.595257 3515 consensus_queue.cc:260] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.595423 3519 log.cc:824] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:13.595539 3515 ts_tablet_manager.cc:1428] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:04:13.595664 3498 heartbeater.cc:503] Master 127.21.66.126:40885 was elected leader, sending a full tablet report...
I20260620 23:04:13.596305 3519 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: No bootstrap required, opened a new log
I20260620 23:04:13.596387 3519 ts_tablet_manager.cc:1397] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:13.597788 3519 raft_consensus.cc:348] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.597936 3519 raft_consensus.cc:374] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:13.597982 3519 raft_consensus.cc:729] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc65870f1c874e969af2065851e597de, State: Initialized, Role: FOLLOWER
I20260620 23:04:13.598097 3519 consensus_queue.cc:260] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.598366 3519 ts_tablet_manager.cc:1428] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:13.598419 3230 heartbeater.cc:503] Master 127.21.66.126:40885 was elected leader, sending a full tablet report...
W20260620 23:04:13.634853 3365 tablet.cc:2369] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:04:13.692085 3231 tablet.cc:2369] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:04:13.795394 3521 raft_consensus.cc:482] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:13.795599 3521 raft_consensus.cc:504] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.796036 3521 leader_election.cc:288] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 348cea95e6314169be2b18059462e979 (127.21.66.66:39931), fc65870f1c874e969af2065851e597de (127.21.66.65:43995)
W20260620 23:04:13.797973 3499 tablet.cc:2369] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:04:13.800237 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.796326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:41791 (local address 127.21.66.66:39931)
0620 23:04:13.796396 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.796399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.796409 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:04:13.796585 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.796586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.796594 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.796637 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:13.796871 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.797626 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.799144 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.799825 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.799917 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.799926 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.800108 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.800156 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:04:13.800202 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.796279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:39931 (local address 127.21.66.67:41791)
0620 23:04:13.796325 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.796336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.796427 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.796666 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.796667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.796669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:13.796754 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.796756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.797701 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.797702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.799002 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.799005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.799922 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.799923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.800010 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.800012 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.800072 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.800122 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":11}
I20260620 23:04:13.800237 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.796512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:45583 (local address 127.21.66.65:43995)
0620 23:04:13.796572 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.796575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.796696 (+ 121us) server_negotiation.cc:407] Connection header received
0620 23:04:13.796739 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.796740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.796749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.796792 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:13.796967 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.797714 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.799171 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.799825 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.799916 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.799926 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.800107 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.800157 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:04:13.800307 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.796428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43995 (local address 127.21.66.67:45583)
0620 23:04:13.796616 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.796626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:13.796687 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:13.796827 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:13.796828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:13.796830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:13.796894 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.796895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.797817 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.797818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.799002 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:13.799005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.799922 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.799923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:13.800006 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.800010 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:04:13.800072 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:04:13.800166 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":6,"thread_start_us":111,"threads_started":1}
I20260620 23:04:13.800624 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ee32751b1e4a1ea8c89d14565a3155" candidate_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348cea95e6314169be2b18059462e979" is_pre_election: true
I20260620 23:04:13.800809 3318 raft_consensus.cc:2454] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7019f5cded4548dfbafcb2d5a2ca6ccd in term 0.
I20260620 23:04:13.800777 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ee32751b1e4a1ea8c89d14565a3155" candidate_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc65870f1c874e969af2065851e597de" is_pre_election: true
I20260620 23:04:13.800952 3184 raft_consensus.cc:2454] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7019f5cded4548dfbafcb2d5a2ca6ccd in term 0.
I20260620 23:04:13.801044 3386 leader_election.cc:302] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348cea95e6314169be2b18059462e979, 7019f5cded4548dfbafcb2d5a2ca6ccd; no voters:
I20260620 23:04:13.801179 3521 raft_consensus.cc:2781] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:04:13.801226 3521 raft_consensus.cc:482] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:04:13.801262 3521 raft_consensus.cc:3037] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:13.801965 3521 raft_consensus.cc:504] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.802099 3521 leader_election.cc:288] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [CANDIDATE]: Term 1 election: Requested vote from peers 348cea95e6314169be2b18059462e979 (127.21.66.66:39931), fc65870f1c874e969af2065851e597de (127.21.66.65:43995)
I20260620 23:04:13.802296 3318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ee32751b1e4a1ea8c89d14565a3155" candidate_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348cea95e6314169be2b18059462e979"
I20260620 23:04:13.802337 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ee32751b1e4a1ea8c89d14565a3155" candidate_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc65870f1c874e969af2065851e597de"
I20260620 23:04:13.802384 3318 raft_consensus.cc:3037] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:13.802407 3184 raft_consensus.cc:3037] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:13.803267 3318 raft_consensus.cc:2454] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7019f5cded4548dfbafcb2d5a2ca6ccd in term 1.
I20260620 23:04:13.803272 3184 raft_consensus.cc:2454] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7019f5cded4548dfbafcb2d5a2ca6ccd in term 1.
I20260620 23:04:13.803449 3386 leader_election.cc:302] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7019f5cded4548dfbafcb2d5a2ca6ccd, fc65870f1c874e969af2065851e597de; no voters:
I20260620 23:04:13.803539 3521 raft_consensus.cc:2781] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:13.803714 3521 raft_consensus.cc:686] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 LEADER]: Becoming Leader. State: Replica: 7019f5cded4548dfbafcb2d5a2ca6ccd, State: Running, Role: LEADER
I20260620 23:04:13.803785 3521 consensus_queue.cc:237] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:13.804613 3039 catalog_manager.cc:5495] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67). New cstate: current_term: 1 leader_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } health_report { overall_health: UNKNOWN } } }
I20260620 23:04:13.810335 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.807797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.21.66.67:33503)
0620 23:04:13.807846 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:13.807848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:13.807908 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:04:13.807971 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:13.807972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:13.807987 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:13.808039 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:04:13.808241 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.808711 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.809569 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:13.809757 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:13.809815 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:13.809954 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:04:13.810048 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:04:13.810217 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:13.810237 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 23:04:13.810265 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:04:13.812090 3184 raft_consensus.cc:1261] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Refusing update from remote peer 7019f5cded4548dfbafcb2d5a2ca6ccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:13.812095 3318 raft_consensus.cc:1261] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Refusing update from remote peer 7019f5cded4548dfbafcb2d5a2ca6ccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:13.812474 3525 consensus_queue.cc:1035] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:13.812626 3521 consensus_queue.cc:1035] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:13.815048 3530 mvcc.cc:205] Tried to move back new op lower bound from 7299058294011506688 to 7299058293980897280. Current Snapshot: MvccSnapshot[applied={T|T < 7299058294011506688}]
I20260620 23:04:13.815474 3529 mvcc.cc:205] Tried to move back new op lower bound from 7299058294011506688 to 7299058293980897280. Current Snapshot: MvccSnapshot[applied={T|T < 7299058294011506688}]
I20260620 23:04:13.817235 3038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:13.817345 3038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:13.818174 3038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Sending DeleteTablet for 3 replicas of tablet 50ee32751b1e4a1ea8c89d14565a3155
I20260620 23:04:13.818519 3432 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ee32751b1e4a1ea8c89d14565a3155 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:13 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:48792
I20260620 23:04:13.818569 3164 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ee32751b1e4a1ea8c89d14565a3155 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:13 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:37934
I20260620 23:04:13.818778 3538 tablet_replica.cc:330] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: stopping tablet replica
I20260620 23:04:13.818770 3298 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ee32751b1e4a1ea8c89d14565a3155 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:13 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:51966
I20260620 23:04:13.818861 3538 raft_consensus.cc:2229] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:04:13.818953 3538 raft_consensus.cc:2258] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:13.818989 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3101
I20260620 23:04:13.818994 3540 tablet_replica.cc:330] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: stopping tablet replica
I20260620 23:04:13.819087 3540 raft_consensus.cc:2229] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:13.819182 3540 raft_consensus.cc:2258] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:13.820345 3538 ts_tablet_manager.cc:1905] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:13.821861 3538 ts_tablet_manager.cc:1918] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:13.821913 3538 log.cc:1196] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/50ee32751b1e4a1ea8c89d14565a3155
I20260620 23:04:13.822158 3538 ts_tablet_manager.cc:1939] T 50ee32751b1e4a1ea8c89d14565a3155 P 7019f5cded4548dfbafcb2d5a2ca6ccd: Deleting consensus metadata
I20260620 23:04:13.822562 3025 catalog_manager.cc:4841] TS 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503): tablet 50ee32751b1e4a1ea8c89d14565a3155 (table test-table [id=a6506aa83b99439e98c2b9edc422b91f]) successfully deleted
I20260620 23:04:13.822669 3540 ts_tablet_manager.cc:1905] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:13.823889 3540 ts_tablet_manager.cc:1918] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:13.823967 3540 log.cc:1196] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/50ee32751b1e4a1ea8c89d14565a3155
I20260620 23:04:13.824223 3540 ts_tablet_manager.cc:1939] T 50ee32751b1e4a1ea8c89d14565a3155 P 348cea95e6314169be2b18059462e979: Deleting consensus metadata
I20260620 23:04:13.824672 3024 catalog_manager.cc:4841] TS 348cea95e6314169be2b18059462e979 (127.21.66.66:39931): tablet 50ee32751b1e4a1ea8c89d14565a3155 (table test-table [id=a6506aa83b99439e98c2b9edc422b91f]) successfully deleted
W20260620 23:04:13.827270 3024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:43995 (error 108)
I20260620 23:04:13.827375 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3235
I20260620 23:04:13.827993 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:13.827741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43995 (local address 127.0.0.1:37940)
0620 23:04:13.827901 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:13.827952 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260620 23:04:13.828136 3024 catalog_manager.cc:4584] TS fc65870f1c874e969af2065851e597de (127.21.66.65:43995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50ee32751b1e4a1ea8c89d14565a3155: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43995: connect: Connection refused (error 111)
I20260620 23:04:13.833791 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3369
I20260620 23:04:13.838989 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3007
I20260620 23:04:13.844467 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40885
--webserver_interface=127.21.66.126
--webserver_port=46771
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:13.928366 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:13.928663 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:13.928706 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:13.929991 3542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:13.930061 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:13.930088 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:13.930109 3542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:13.930127 3542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:13.931669 3542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=46771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:13.931959 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:13.932224 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:13.941874 3548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:13.941890 3549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:13.941943 3542 server_base.cc:1034] running on GCE node
W20260620 23:04:13.941881 3551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:13.942479 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:13.942775 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:13.943953 3542 hybrid_clock.cc:648] HybridClock initialized: now 1781996653943929 us; error 37 us; skew 500 ppm
Jun 20 23:04:13 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:13.946574 3542 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:13.947055 3542 webserver.cc:460] Webserver started at http://127.21.66.126:46771/ using document root <none> and password file <none>
I20260620 23:04:13.947314 3542 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:13.947372 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:13.948876 3542 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 23:04:13.949774 3558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:13.949972 3542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:13.950085 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "254a17821b47481ab4cf54c9e72e8938"
format_stamp: "Formatted at 2026-06-20 23:04:12 on dist-test-slave-qgtk"
server_key: "5c8fd9afb1bb8ddf037683acda8d6cd0"
server_key_iv: "a6e77f7338aa28ec5332f2472223f92e"
server_key_version: "encryptionkey@0"
I20260620 23:04:13.950557 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:13.959007 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:13.961374 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:13.961581 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:13.966384 3542 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:40885
I20260620 23:04:13.966414 3610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:40885 every 8 connection(s)
I20260620 23:04:13.966897 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:13.968669 3611 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:04:13.969332 3611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Bootstrap starting.
I20260620 23:04:13.969758 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3542
I20260620 23:04:13.970048 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:43995
--local_ip_for_outbound_sockets=127.21.66.65
--tserver_master_addrs=127.21.66.126:40885
--webserver_port=35515
--webserver_interface=127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:13.977679 3611 log.cc:824] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:13.979784 3611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:04:13.980123 3611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Bootstrap complete.
I20260620 23:04:13.982597 3611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:13.982868 3611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a17821b47481ab4cf54c9e72e8938, State: Initialized, Role: FOLLOWER
I20260620 23:04:13.983052 3611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:13.983135 3611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:13.983192 3611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:13.983246 3611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:04:13.984341 3611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:13.984462 3611 leader_election.cc:302] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254a17821b47481ab4cf54c9e72e8938; no voters:
I20260620 23:04:13.984656 3611 leader_election.cc:288] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:04:13.984735 3615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:04:13.984897 3615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 LEADER]: Becoming Leader. State: Replica: 254a17821b47481ab4cf54c9e72e8938, State: Running, Role: LEADER
I20260620 23:04:13.984933 3611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:13.985008 3615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:13.985292 3615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254a17821b47481ab4cf54c9e72e8938. Latest consensus state: current_term: 2 leader_uuid: "254a17821b47481ab4cf54c9e72e8938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } } }
I20260620 23:04:13.985476 3615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:13.985294 3616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "254a17821b47481ab4cf54c9e72e8938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } } }
I20260620 23:04:13.985740 3616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:13.986877 3631 catalog_manager.cc:1259] Loaded cluster ID: b85089f78dfd4f998a5118723defacb8
I20260620 23:04:13.986929 3631 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: loading cluster ID for follower catalog manager: success
W20260620 23:04:13.987123 3631 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260620 23:04:13.987166 3631 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260620 23:04:13.987210 3625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:13.987558 3625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:13.987649 3625 catalog_manager.cc:1259] Loaded cluster ID: b85089f78dfd4f998a5118723defacb8
I20260620 23:04:13.987681 3625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:13.987830 3625 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 23:04:13.992527 3625 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 23:04:13.995240 3625 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:13.995462 3625 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Loaded TSK: 0
I20260620 23:04:13.995703 3625 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:04:14.055768 3613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:14.056016 3613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:14.056066 3613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:14.057387 3613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:14.057467 3613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:14.057514 3613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:04:14.059110 3613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:43995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=35515
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:14.059391 3613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:14.059638 3613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:14.064905 3639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.064905 3638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.064931 3613 server_base.cc:1034] running on GCE node
W20260620 23:04:14.065065 3641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.065485 3613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:14.065799 3613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:14.067011 3613 hybrid_clock.cc:648] HybridClock initialized: now 1781996654066991 us; error 30 us; skew 500 ppm
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:14.069695 3613 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:04:14.070271 3613 webserver.cc:460] Webserver started at http://127.21.66.65:35515/ using document root <none> and password file <none>
I20260620 23:04:14.070533 3613 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:14.070591 3613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:14.072012 3613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:14.072818 3648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.072965 3613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:14.073066 3613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "fc65870f1c874e969af2065851e597de"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "faf4153f449cd1f3d4b58d654893a018"
server_key_iv: "33bcb0f28a4f6855312a210c375493ef"
server_key_version: "encryptionkey@0"
I20260620 23:04:14.073484 3613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:14.096447 3613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:14.099088 3613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:14.099211 3613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:14.099545 3613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:14.100097 3655 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:04:14.101574 3613 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:04:14.101634 3613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:04:14.101667 3613 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:04:14.102430 3613 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:04:14.102473 3613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:14.102526 3655 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Bootstrap starting.
I20260620 23:04:14.110342 3613 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:43995
I20260620 23:04:14.110570 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:43995 every 8 connection(s)
I20260620 23:04:14.110941 3613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:14.114302 3655 log.cc:824] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:14.115657 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3613
I20260620 23:04:14.115948 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:39931
--local_ip_for_outbound_sockets=127.21.66.66
--tserver_master_addrs=127.21.66.126:40885
--webserver_port=45463
--webserver_interface=127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:14.117977 3655 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:04:14.118350 3655 tablet_bootstrap.cc:492] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Bootstrap complete.
I20260620 23:04:14.119474 3655 ts_tablet_manager.cc:1397] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
I20260620 23:04:14.121021 3655 raft_consensus.cc:348] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.121366 3655 raft_consensus.cc:729] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc65870f1c874e969af2065851e597de, State: Initialized, Role: FOLLOWER
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:14.121549 3655 consensus_queue.cc:260] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.122906 3655 ts_tablet_manager.cc:1428] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:04:14.126451 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.111882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:45711 (local address 127.21.66.126:40885)
0620 23:04:14.114297 (+ 2415us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.114309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.114419 (+ 110us) server_negotiation.cc:407] Connection header received
0620 23:04:14.114589 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.114597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.114661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.114828 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:14.115531 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.117347 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.119297 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.119898 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.120546 (+ 648us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.122687 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:14.122691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:14.122691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:14.122704 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:14.124586 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.125033 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.125037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.125037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.125144 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.125421 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.125422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.125422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.125669 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:14.125763 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.126076 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.126241 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2351,"thread_start_us":84,"threads_started":1}
I20260620 23:04:14.126451 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.111692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:45711)
0620 23:04:14.111908 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.111970 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.112632 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.114888 (+ 2256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.114892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.115164 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:14.115358 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.115363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.117536 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.117537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.119192 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.119197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.120050 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.120051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.120211 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.120889 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:14.120899 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:14.122560 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:14.124696 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.124703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.124705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.124943 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.125224 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.125225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.125225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.125346 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.125756 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:14.125763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:14.125905 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.126017 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.126245 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
I20260620 23:04:14.127192 3763 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40885
I20260620 23:04:14.127305 3763 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:14.127491 3763 heartbeater.cc:511] Master 127.21.66.126:40885 requested a full tablet report, sending...
I20260620 23:04:14.128270 3575 ts_manager.cc:194] Registered new tserver with Master: fc65870f1c874e969af2065851e597de (127.21.66.65:43995)
I20260620 23:04:14.129618 3575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:45711
I20260620 23:04:14.130512 3763 heartbeater.cc:503] Master 127.21.66.126:40885 was elected leader, sending a full tablet report...
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996653, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.65@KRBTEST.COM
I20260620 23:04:14.138497 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.128882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43995 (local address 127.0.0.1:37942)
0620 23:04:14.129305 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.129343 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.129407 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.129816 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.129820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.130991 (+ 1171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:14.131115 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.131118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.131995 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.131996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.132581 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.132583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.133076 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.133078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.133163 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.133703 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:14.133708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:14.135132 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:14.137205 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.137212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.137215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.137427 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.137681 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.137682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.137683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.137760 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.138068 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:14.138075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:14.138152 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.138256 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.138388 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":100,"threads_started":1}
I20260620 23:04:14.138497 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.129151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.21.66.65:43995)
0620 23:04:14.129519 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.129524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.129547 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:04:14.129579 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.129581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.129595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.129657 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:14.131208 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.131862 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.132685 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.132991 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.133077 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.135238 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:14.135246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:14.135246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:14.135254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:14.137010 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.137517 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.137521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.137522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.137590 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.137827 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.137828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.137829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.137988 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:14.138077 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.138295 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.138388 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":72,"threads_started":1}
I20260620 23:04:14.139154 3697 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ee32751b1e4a1ea8c89d14565a3155 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:13 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:37942
I20260620 23:04:14.139691 3772 tablet_replica.cc:330] stopping tablet replica
I20260620 23:04:14.139814 3772 raft_consensus.cc:2229] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:14.139907 3772 raft_consensus.cc:2258] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:14.140223 3772 ts_tablet_manager.cc:1905] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:14.142431 3772 ts_tablet_manager.cc:1918] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:14.142570 3772 log.cc:1196] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/50ee32751b1e4a1ea8c89d14565a3155
I20260620 23:04:14.142839 3772 ts_tablet_manager.cc:1939] T 50ee32751b1e4a1ea8c89d14565a3155 P fc65870f1c874e969af2065851e597de: Deleting consensus metadata
I20260620 23:04:14.143394 3560 catalog_manager.cc:4841] TS fc65870f1c874e969af2065851e597de (127.21.66.65:43995): tablet 50ee32751b1e4a1ea8c89d14565a3155 (table test-table [id=a6506aa83b99439e98c2b9edc422b91f]) successfully deleted
W20260620 23:04:14.211905 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:14.212177 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:14.212206 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:14.213610 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:14.213660 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:14.213692 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:04:14.215420 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:39931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=45463
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:14.215751 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:14.216027 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:14.221540 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.221549 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.221552 3780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.221935 3768 server_base.cc:1034] running on GCE node
I20260620 23:04:14.222213 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:14.222484 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:14.223694 3768 hybrid_clock.cc:648] HybridClock initialized: now 1781996654223675 us; error 29 us; skew 500 ppm
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:14.226322 3768 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:04:14.226845 3768 webserver.cc:460] Webserver started at http://127.21.66.66:45463/ using document root <none> and password file <none>
I20260620 23:04:14.227085 3768 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:14.227190 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:14.228557 3768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:04:14.229185 3787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.229386 3768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:04:14.229490 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "348cea95e6314169be2b18059462e979"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "c1ecd62e3eb09e626aaa78875ab0bbb4"
server_key_iv: "5bc159ff091b959922d56c106fe71819"
server_key_version: "encryptionkey@0"
I20260620 23:04:14.229863 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:14.250075 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:14.252630 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:14.252808 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:14.253075 3768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:14.253396 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:14.253463 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.253497 3768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:14.253510 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.259863 3768 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:39931
I20260620 23:04:14.259900 3900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:39931 every 8 connection(s)
I20260620 23:04:14.260250 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:04:14.264062 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3768
I20260620 23:04:14.264452 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:33503
--local_ip_for_outbound_sockets=127.21.66.67
--tserver_master_addrs=127.21.66.126:40885
--webserver_port=33297
--webserver_interface=127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:14.271636 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.260945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:51003)
0620 23:04:14.261110 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.261161 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.262107 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.262397 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.262401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.262674 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:14.262869 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.262873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.263571 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.263572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.264465 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.264468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.265084 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.265084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.265245 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.265808 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:14.265817 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:14.267627 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:14.269783 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.269790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.269792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.270022 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.270308 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.270309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.270310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.270416 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.271000 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:14.271007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:14.271170 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.271299 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.271448 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260620 23:04:14.271690 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.261157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:51003 (local address 127.21.66.126:40885)
0620 23:04:14.261217 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.261219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.261231 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:04:14.262260 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.262261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.262273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.262326 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:14.262968 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.263468 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.264580 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.264971 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.265060 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.267740 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:14.267741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:14.267742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:14.267751 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:14.269672 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.270140 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.270142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.270142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.270219 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.270605 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.270706 (+ 101us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.270707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.270890 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:14.270986 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.271362 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.271479 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:04:14.272619 3901 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40885
I20260620 23:04:14.272746 3901 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:14.272993 3901 heartbeater.cc:511] Master 127.21.66.126:40885 requested a full tablet report, sending...
I20260620 23:04:14.273468 3575 ts_manager.cc:194] Registered new tserver with Master: 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
I20260620 23:04:14.273941 3575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:51003
W20260620 23:04:14.360298 3904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:14.360529 3904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:14.360558 3904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:14.361955 3904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:14.362010 3904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:14.362042 3904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:04:14.363730 3904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:33503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=33297
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:14.364004 3904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:14.364264 3904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:14.369990 3909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.369982 3910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.370234 3904 server_base.cc:1034] running on GCE node
W20260620 23:04:14.369982 3912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.370566 3904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:14.370827 3904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:14.372010 3904 hybrid_clock.cc:648] HybridClock initialized: now 1781996654372010 us; error 49 us; skew 500 ppm
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:14.374974 3904 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:04:14.375535 3904 webserver.cc:460] Webserver started at http://127.21.66.67:33297/ using document root <none> and password file <none>
I20260620 23:04:14.375746 3904 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:14.375804 3904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:14.377478 3904 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:04:14.378360 3919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.378518 3904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:04:14.378662 3904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "05c5b3073fcadf456e616d188329f6ed"
server_key_iv: "c5c2265f58cd538189377c000973f557"
server_key_version: "encryptionkey@0"
I20260620 23:04:14.378998 3904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:14.389722 3904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:14.391996 3904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:14.392112 3904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:14.392304 3904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:14.392555 3904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:14.392592 3904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.392614 3904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:14.392627 3904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.398332 3904 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:33503
I20260620 23:04:14.398419 4032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:33503 every 8 connection(s)
I20260620 23:04:14.398756 3904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:04:14.400735 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 3904
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:14.408550 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.401188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.21.66.126:40885)
0620 23:04:14.401326 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.401328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.401337 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:14.401401 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.401401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.401412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.401468 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:14.402163 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.402853 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.403691 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.403926 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.404000 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.405307 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:14.405309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:14.405309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:14.405318 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:14.407200 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.407678 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.407679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.407679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.407737 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.407947 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.407947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.407948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.408085 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:14.408244 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.408331 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.408430 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260620 23:04:14.409844 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.399534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:44577 (local address 127.21.66.126:40885)
0620 23:04:14.399637 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.399639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.399812 (+ 173us) server_negotiation.cc:407] Connection header received
0620 23:04:14.400445 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.400446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.400461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.400514 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:14.401768 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.402515 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.403754 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.403934 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.404016 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.406406 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:14.406407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:14.406408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:14.406415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:14.408225 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.408754 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.408755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.408756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.408812 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:14.409132 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:14.409132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:14.409133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:14.409244 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:14.409323 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.409668 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.409747 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:04:14.410076 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.399461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:44577)
0620 23:04:14.399641 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.399705 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.400343 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.400621 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.400627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.401452 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:14.401672 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.401677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.402606 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.402607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.403624 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.403627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.404039 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.404040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.404243 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.404771 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:14.404779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:14.406319 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:14.408313 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.408321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.408324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.408649 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.408888 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:14.408888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:14.408888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:14.409053 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:14.409315 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:14.409321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:14.409525 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.409647 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.409826 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260620 23:04:14.410570 4033 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40885
I20260620 23:04:14.410669 4033 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:14.410864 4033 heartbeater.cc:511] Master 127.21.66.126:40885 requested a full tablet report, sending...
I20260620 23:04:14.411303 3575 ts_manager.cc:194] Registered new tserver with Master: 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503)
I20260620 23:04:14.411715 3575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:44577
I20260620 23:04:14.420763 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:04:14.424252 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.421245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.21.66.126:40885)
0620 23:04:14.421306 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.421311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.421339 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:04:14.421414 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.421415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.421452 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.421514 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:04:14.421705 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.422420 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.423321 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.423528 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.423593 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.423775 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:04:14.423891 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:04:14.423971 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.424051 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.424152 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:04:14.424590 3575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:04:14.426275 3575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:04:14.428917 3697 tablet_service.cc:1467] Processing CreateTablet for tablet 29bd071aefb84d3ba055538b749465bf (DEFAULT_TABLE table=test-table [id=ed73df3b8d6a4cf096254ded73d37672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:14.429271 3697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bd071aefb84d3ba055538b749465bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:14.430819 3655 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Bootstrap starting.
I20260620 23:04:14.431140 3655 tablet_bootstrap.cc:654] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:14.431887 3655 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: No bootstrap required, opened a new log
I20260620 23:04:14.431938 3655 ts_tablet_manager.cc:1397] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:14.432080 3655 raft_consensus.cc:348] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.432155 3655 raft_consensus.cc:374] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:14.432201 3655 raft_consensus.cc:729] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc65870f1c874e969af2065851e597de, State: Initialized, Role: FOLLOWER
I20260620 23:04:14.432243 3655 consensus_queue.cc:260] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.432343 3655 ts_tablet_manager.cc:1428] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.432854 4039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.428412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:39931 (local address 127.0.0.1:51974)
0620 23:04:14.428762 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.428772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.428825 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.429067 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.429069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.429071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:14.429224 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.429227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.430287 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.430288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.431502 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.431505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.432512 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.432514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.432586 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.432610 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.432672 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.432726 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":14,"thread_start_us":119,"threads_started":1}
I20260620 23:04:14.432839 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.428682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.21.66.66:39931)
0620 23:04:14.428896 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.428902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.428911 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:14.428954 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.428956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.428969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.429060 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:14.429354 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.430169 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.431592 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.432422 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.432511 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.432577 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.432701 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.432746 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":127,"threads_started":1}
I20260620 23:04:14.433040 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.428412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33503 (local address 127.0.0.1:48812)
0620 23:04:14.428482 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.428495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.428558 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.429106 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.429107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.429109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:14.429156 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.429158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.430492 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.430493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.431563 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.431565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.432780 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.432781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.432848 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.432853 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.432883 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.432934 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 23:04:14.433341 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.428573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.21.66.67:33503)
0620 23:04:14.428931 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.428936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.428946 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:04:14.428986 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.428988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.429003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.429239 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:14.429311 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.430386 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.431895 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.432695 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.433157 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.433229 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.433253 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.433285 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":147,"threads_started":1}
I20260620 23:04:14.433738 3835 tablet_service.cc:1467] Processing CreateTablet for tablet 29bd071aefb84d3ba055538b749465bf (DEFAULT_TABLE table=test-table [id=ed73df3b8d6a4cf096254ded73d37672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:14.434021 3967 tablet_service.cc:1467] Processing CreateTablet for tablet 29bd071aefb84d3ba055538b749465bf (DEFAULT_TABLE table=test-table [id=ed73df3b8d6a4cf096254ded73d37672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:14.434217 3835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bd071aefb84d3ba055538b749465bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:14.434392 3967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29bd071aefb84d3ba055538b749465bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:14.437049 4043 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Bootstrap starting.
I20260620 23:04:14.437820 4043 tablet_bootstrap.cc:654] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:14.438319 4044 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Bootstrap starting.
I20260620 23:04:14.438452 4043 log.cc:824] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:14.438997 4044 tablet_bootstrap.cc:654] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:14.439324 4043 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: No bootstrap required, opened a new log
I20260620 23:04:14.439411 4043 ts_tablet_manager.cc:1397] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:14.439559 4044 log.cc:824] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:14.440488 4044 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: No bootstrap required, opened a new log
I20260620 23:04:14.440558 4044 ts_tablet_manager.cc:1397] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:14.441437 4043 raft_consensus.cc:348] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.441617 4043 raft_consensus.cc:374] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:14.441653 4043 raft_consensus.cc:729] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7019f5cded4548dfbafcb2d5a2ca6ccd, State: Initialized, Role: FOLLOWER
I20260620 23:04:14.441774 4043 consensus_queue.cc:260] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.442058 4043 ts_tablet_manager.cc:1428] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:04:14.442075 4033 heartbeater.cc:503] Master 127.21.66.126:40885 was elected leader, sending a full tablet report...
I20260620 23:04:14.442569 4044 raft_consensus.cc:348] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.442696 4044 raft_consensus.cc:374] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:14.442718 4044 raft_consensus.cc:729] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348cea95e6314169be2b18059462e979, State: Initialized, Role: FOLLOWER
I20260620 23:04:14.442826 4044 consensus_queue.cc:260] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.443106 4044 ts_tablet_manager.cc:1428] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:04:14.443135 3901 heartbeater.cc:503] Master 127.21.66.126:40885 was elected leader, sending a full tablet report...
I20260620 23:04:14.494925 3769 raft_consensus.cc:482] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:14.495080 3769 raft_consensus.cc:504] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.495500 3769 leader_election.cc:288] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503), 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
I20260620 23:04:14.499189 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.495547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33503 (local address 127.21.66.65:38147)
0620 23:04:14.495635 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.495647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.495705 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.495899 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.495900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.495902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:14.495994 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.495996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.496760 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.496761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.498054 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.498057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.498864 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.498865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.498953 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.498983 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.499047 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.499094 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":30}
I20260620 23:04:14.499230 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.495597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:38147 (local address 127.21.66.67:33503)
0620 23:04:14.495668 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.495671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.495683 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:04:14.495817 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.495818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.495829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.495902 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:14.496144 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.496659 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.498180 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.498772 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.498852 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.498860 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.499116 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.499160 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:04:14.499230 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.495792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:47005 (local address 127.21.66.66:39931)
0620 23:04:14.495831 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.495833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.495996 (+ 163us) server_negotiation.cc:407] Connection header received
0620 23:04:14.496033 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.496034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.496053 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.496111 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:14.496259 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.496716 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.498180 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.498772 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.498853 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.498861 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.499115 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.499159 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:04:14.499200 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.495731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:39931 (local address 127.21.66.65:47005)
0620 23:04:14.495916 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.495926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.495986 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.496113 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:14.496114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:14.496115 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:14.496190 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.496192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.496786 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.496786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.498066 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:14.498068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.498888 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.498889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:14.498953 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.498983 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:04:14.499047 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 23:04:14.499130 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":5,"thread_start_us":118,"threads_started":1}
I20260620 23:04:14.499612 3987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bd071aefb84d3ba055538b749465bf" candidate_uuid: "fc65870f1c874e969af2065851e597de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" is_pre_election: true
I20260620 23:04:14.499804 3987 raft_consensus.cc:2454] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc65870f1c874e969af2065851e597de in term 0.
I20260620 23:04:14.499796 3855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bd071aefb84d3ba055538b749465bf" candidate_uuid: "fc65870f1c874e969af2065851e597de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348cea95e6314169be2b18059462e979" is_pre_election: true
I20260620 23:04:14.499930 3855 raft_consensus.cc:2454] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc65870f1c874e969af2065851e597de in term 0.
I20260620 23:04:14.500053 3651 leader_election.cc:302] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7019f5cded4548dfbafcb2d5a2ca6ccd, fc65870f1c874e969af2065851e597de; no voters:
I20260620 23:04:14.500200 3769 raft_consensus.cc:2781] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:04:14.500284 3769 raft_consensus.cc:482] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:04:14.500320 3769 raft_consensus.cc:3037] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:14.501086 3769 raft_consensus.cc:504] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.501243 3769 leader_election.cc:288] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 1 election: Requested vote from peers 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503), 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
I20260620 23:04:14.501420 3855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bd071aefb84d3ba055538b749465bf" candidate_uuid: "fc65870f1c874e969af2065851e597de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348cea95e6314169be2b18059462e979"
I20260620 23:04:14.501435 3987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29bd071aefb84d3ba055538b749465bf" candidate_uuid: "fc65870f1c874e969af2065851e597de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd"
I20260620 23:04:14.501523 3987 raft_consensus.cc:3037] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:14.501529 3855 raft_consensus.cc:3037] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:14.502395 3987 raft_consensus.cc:2454] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc65870f1c874e969af2065851e597de in term 1.
I20260620 23:04:14.502418 3855 raft_consensus.cc:2454] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc65870f1c874e969af2065851e597de in term 1.
I20260620 23:04:14.502595 3650 leader_election.cc:302] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 348cea95e6314169be2b18059462e979, fc65870f1c874e969af2065851e597de; no voters:
I20260620 23:04:14.502719 3769 raft_consensus.cc:2781] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:14.502887 3769 raft_consensus.cc:686] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 LEADER]: Becoming Leader. State: Replica: fc65870f1c874e969af2065851e597de, State: Running, Role: LEADER
I20260620 23:04:14.502971 3769 consensus_queue.cc:237] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.503866 3575 catalog_manager.cc:5495] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de reported cstate change: term changed from 0 to 1, leader changed from <none> to fc65870f1c874e969af2065851e597de (127.21.66.65). New cstate: current_term: 1 leader_uuid: "fc65870f1c874e969af2065851e597de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } health_report { overall_health: HEALTHY } } }
W20260620 23:04:14.510743 3902 tablet.cc:2369] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:04:14.511011 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.508272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.21.66.65:43995)
0620 23:04:14.508340 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:14.508342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:14.508351 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:14.508386 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:14.508386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:14.508411 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:14.508488 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:04:14.508632 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.509193 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.509998 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:14.510307 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:14.510385 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:14.510549 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:04:14.510693 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:04:14.510781 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:14.510877 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 23:04:14.510903 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:04:14.512696 3855 raft_consensus.cc:1261] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Refusing update from remote peer fc65870f1c874e969af2065851e597de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:14.512696 3987 raft_consensus.cc:1261] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 FOLLOWER]: Refusing update from remote peer fc65870f1c874e969af2065851e597de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:14.513115 3769 consensus_queue.cc:1035] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:14.513302 4050 consensus_queue.cc:1035] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [LEADER]: Connected to new peer: Peer: permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:14.516175 4053 mvcc.cc:205] Tried to move back new op lower bound from 7299058296881201152 to 7299058296845074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299058296881201152}]
I20260620 23:04:14.516707 4054 mvcc.cc:205] Tried to move back new op lower bound from 7299058296881201152 to 7299058296845074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299058296881201152}]
I20260620 23:04:14.517004 4055 mvcc.cc:205] Tried to move back new op lower bound from 7299058296881201152 to 7299058296845074432. Current Snapshot: MvccSnapshot[applied={T|T < 7299058296881201152}]
I20260620 23:04:14.518745 3575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:14.518859 3575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:14.519652 3575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Sending DeleteTablet for 3 replicas of tablet 29bd071aefb84d3ba055538b749465bf
I20260620 23:04:14.520004 3697 tablet_service.cc:1514] Processing DeleteTablet for tablet 29bd071aefb84d3ba055538b749465bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:14 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:37942
I20260620 23:04:14.520007 3835 tablet_service.cc:1514] Processing DeleteTablet for tablet 29bd071aefb84d3ba055538b749465bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:14 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:51974
I20260620 23:04:14.520030 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3613
I20260620 23:04:14.520176 3967 tablet_service.cc:1514] Processing DeleteTablet for tablet 29bd071aefb84d3ba055538b749465bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:14 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:48812
I20260620 23:04:14.520364 3772 tablet_replica.cc:330] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: stopping tablet replica
I20260620 23:04:14.520419 3772 raft_consensus.cc:2229] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:04:14.520501 3772 raft_consensus.cc:2258] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:14.520720 4063 tablet_replica.cc:330] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: stopping tablet replica
I20260620 23:04:14.520841 4063 raft_consensus.cc:2229] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:14.520977 4063 raft_consensus.cc:2258] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:14.521414 4063 ts_tablet_manager.cc:1905] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:14.523308 4063 ts_tablet_manager.cc:1918] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:14.523404 4063 log.cc:1196] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/29bd071aefb84d3ba055538b749465bf
I20260620 23:04:14.523733 4063 ts_tablet_manager.cc:1939] T 29bd071aefb84d3ba055538b749465bf P 348cea95e6314169be2b18059462e979: Deleting consensus metadata
I20260620 23:04:14.524343 3560 catalog_manager.cc:4841] TS 348cea95e6314169be2b18059462e979 (127.21.66.66:39931): tablet 29bd071aefb84d3ba055538b749465bf (table test-table [id=ed73df3b8d6a4cf096254ded73d37672]) successfully deleted
I20260620 23:04:14.526289 4064 tablet_replica.cc:330] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: stopping tablet replica
I20260620 23:04:14.526454 4064 raft_consensus.cc:2229] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:14.526595 4064 raft_consensus.cc:2258] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 23:04:14.526968 3560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:43995 (error 108)
I20260620 23:04:14.527302 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3768
I20260620 23:04:14.527385 4064 ts_tablet_manager.cc:1905] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:14.527971 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.527403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43995 (local address 127.0.0.1:37954)
0620 23:04:14.527876 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.527927 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444}
W20260620 23:04:14.528223 3560 catalog_manager.cc:4584] TS fc65870f1c874e969af2065851e597de (127.21.66.65:43995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29bd071aefb84d3ba055538b749465bf: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43995: connect: Connection refused (error 111)
I20260620 23:04:14.529273 4064 ts_tablet_manager.cc:1918] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:14.529340 4064 log.cc:1196] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/29bd071aefb84d3ba055538b749465bf
I20260620 23:04:14.529651 4064 ts_tablet_manager.cc:1939] T 29bd071aefb84d3ba055538b749465bf P 7019f5cded4548dfbafcb2d5a2ca6ccd: Deleting consensus metadata
I20260620 23:04:14.530076 3561 catalog_manager.cc:4841] TS 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503): tablet 29bd071aefb84d3ba055538b749465bf (table test-table [id=ed73df3b8d6a4cf096254ded73d37672]) successfully deleted
I20260620 23:04:14.533166 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3904
I20260620 23:04:14.539297 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 3542
I20260620 23:04:14.545486 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40885
--webserver_interface=127.21.66.126
--webserver_port=46771
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:40885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:14.632097 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:14.632359 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:14.632398 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:14.633543 4066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:14.633606 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:14.633628 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:14.633649 4066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:14.633678 4066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:14.635228 4066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:40885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=46771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:14.635497 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:14.635756 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:14.640851 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.640905 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.640851 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.641426 4066 server_base.cc:1034] running on GCE node
I20260620 23:04:14.641727 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:14.642035 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:14.643234 4066 hybrid_clock.cc:648] HybridClock initialized: now 1781996654643233 us; error 59 us; skew 500 ppm
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:14.645812 4066 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:14.646368 4066 webserver.cc:460] Webserver started at http://127.21.66.126:46771/ using document root <none> and password file <none>
I20260620 23:04:14.646600 4066 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:14.646665 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:14.647979 4066 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:14.648666 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.648834 4066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:04:14.648928 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "254a17821b47481ab4cf54c9e72e8938"
format_stamp: "Formatted at 2026-06-20 23:04:12 on dist-test-slave-qgtk"
server_key: "5c8fd9afb1bb8ddf037683acda8d6cd0"
server_key_iv: "a6e77f7338aa28ec5332f2472223f92e"
server_key_version: "encryptionkey@0"
I20260620 23:04:14.649298 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:14.657503 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:14.659668 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:14.659839 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:14.664188 4066 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:40885
I20260620 23:04:14.664228 4133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:40885 every 8 connection(s)
I20260620 23:04:14.664590 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:14.666103 4134 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:04:14.666858 4134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Bootstrap starting.
I20260620 23:04:14.670495 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4066
I20260620 23:04:14.670836 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:43995
--local_ip_for_outbound_sockets=127.21.66.65
--tserver_master_addrs=127.21.66.126:40885
--webserver_port=35515
--webserver_interface=127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:14.675741 4134 log.cc:824] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:14.679296 4134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:04:14.679689 4134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938: Bootstrap complete.
I20260620 23:04:14.683399 4134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:14.683727 4134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254a17821b47481ab4cf54c9e72e8938, State: Initialized, Role: FOLLOWER
I20260620 23:04:14.683887 4134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:14.683952 4134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:14.683996 4134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:14.684043 4134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:04:14.685362 4134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:14.685470 4134 leader_election.cc:302] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254a17821b47481ab4cf54c9e72e8938; no voters:
I20260620 23:04:14.685645 4134 leader_election.cc:288] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:04:14.685724 4138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:04:14.685887 4134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:14.685915 4138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [term 3 LEADER]: Becoming Leader. State: Replica: 254a17821b47481ab4cf54c9e72e8938, State: Running, Role: LEADER
I20260620 23:04:14.686035 4138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } }
I20260620 23:04:14.686367 4138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254a17821b47481ab4cf54c9e72e8938. Latest consensus state: current_term: 3 leader_uuid: "254a17821b47481ab4cf54c9e72e8938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } } }
I20260620 23:04:14.686455 4138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:14.686610 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "254a17821b47481ab4cf54c9e72e8938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254a17821b47481ab4cf54c9e72e8938" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40885 } } }
I20260620 23:04:14.686676 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254a17821b47481ab4cf54c9e72e8938 [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:14.687036 4146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:14.687975 4146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:14.688122 4146 catalog_manager.cc:1259] Loaded cluster ID: b85089f78dfd4f998a5118723defacb8
I20260620 23:04:14.688171 4146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 23:04:14.688421 4146 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc0c9c4c9ad google::LogMessage::Fail() at ??:0
@ 0x7fc0c9c4f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc0c9c4c4de google::LogMessage::Flush() at ??:0
@ 0x7fc0c9c4da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc0d1b6d0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc0d1b768b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc0cbb1a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc0cbb0f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc0cc9736db start_thread at ??:0
@ 0x7fc0c874a61f clone at ??:0
W20260620 23:04:14.773213 4136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:14.773461 4136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:14.773502 4136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:14.774773 4136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:14.774837 4136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:14.774883 4136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:04:14.776489 4136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:43995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=35515
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:14.776772 4136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:14.777001 4136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:14.782352 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.782392 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.782387 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.782636 4136 server_base.cc:1034] running on GCE node
I20260620 23:04:14.782853 4136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:14.783140 4136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:14.784324 4136 hybrid_clock.cc:648] HybridClock initialized: now 1781996654784309 us; error 26 us; skew 500 ppm
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:14.787340 4136 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:04:14.787928 4136 webserver.cc:460] Webserver started at http://127.21.66.65:35515/ using document root <none> and password file <none>
I20260620 23:04:14.788194 4136 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:14.788261 4136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:14.790028 4136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:04:14.790812 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.791006 4136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:14.791103 4136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "fc65870f1c874e969af2065851e597de"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "faf4153f449cd1f3d4b58d654893a018"
server_key_iv: "33bcb0f28a4f6855312a210c375493ef"
server_key_version: "encryptionkey@0"
I20260620 23:04:14.791589 4136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:14.823540 4136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:14.826057 4136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:14.826242 4136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:14.826514 4136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:14.827016 4187 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:04:14.828469 4136 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:04:14.828537 4136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:04:14.828573 4136 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:04:14.829414 4136 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:04:14.829459 4136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:04:14.829548 4187 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Bootstrap starting.
I20260620 23:04:14.836637 4136 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:43995
I20260620 23:04:14.836761 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:43995 every 8 connection(s)
I20260620 23:04:14.837157 4136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:14.840158 4187 log.cc:824] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:14.842286 4187 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:04:14.842556 4187 tablet_bootstrap.cc:492] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Bootstrap complete.
I20260620 23:04:14.843468 4187 ts_tablet_manager.cc:1397] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260620 23:04:14.844491 4187 raft_consensus.cc:348] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.844732 4187 raft_consensus.cc:729] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc65870f1c874e969af2065851e597de, State: Initialized, Role: FOLLOWER
I20260620 23:04:14.844905 4187 consensus_queue.cc:260] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:14.845189 4187 ts_tablet_manager.cc:1428] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:04:14.847282 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4136
I20260620 23:04:14.847568 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:39931
--local_ip_for_outbound_sockets=127.21.66.66
--tserver_master_addrs=127.21.66.126:40885
--webserver_port=45463
--webserver_interface=127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:14.893836 4297 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:04:14.837897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:55969)
0620 23:04:14.838150 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.838199 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:14.838997 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:14.893616 (+ 54619us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":117,"threads_started":1}
W20260620 23:04:14.894174 4295 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:40885 (0 consecutive failures): Network error: Failed to ping master at 127.21.66.126:40885: Client connection negotiation failed: client connection to 127.21.66.126:40885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:04:14.894583 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.894458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:56041)
0620 23:04:14.894517 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.894562 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:04:14.894971 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.894854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:51421)
0620 23:04:14.894902 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.894928 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:04:14.895097 4295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:04:14.943866 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:14.944128 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:14.944175 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:14.945482 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:14.945557 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:14.945613 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:04:14.947337 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:39931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=45463
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:14.947644 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:14.947901 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:14.953528 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.953599 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:14.953590 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:14.953756 4300 server_base.cc:1034] running on GCE node
I20260620 23:04:14.953979 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:14.954296 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:14.955502 4300 hybrid_clock.cc:648] HybridClock initialized: now 1781996654955485 us; error 27 us; skew 500 ppm
Jun 20 23:04:14 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:14.958253 4300 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:04:14.958725 4300 webserver.cc:460] Webserver started at http://127.21.66.66:45463/ using document root <none> and password file <none>
I20260620 23:04:14.958976 4300 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:14.959115 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:14.960565 4300 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:04:14.961272 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.961458 4300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:14.961558 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "348cea95e6314169be2b18059462e979"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "c1ecd62e3eb09e626aaa78875ab0bbb4"
server_key_iv: "5bc159ff091b959922d56c106fe71819"
server_key_version: "encryptionkey@0"
I20260620 23:04:14.961934 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:14.974035 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:14.976421 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:14.976606 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:14.976853 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:14.977174 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:14.977211 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.977236 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:14.977247 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:14.983192 4300 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:39931
I20260620 23:04:14.983301 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:39931 every 8 connection(s)
I20260620 23:04:14.983626 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:04:14.984781 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.984364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:48945)
0620 23:04:14.984514 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.984603 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
W20260620 23:04:14.985216 4429 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:40885 (0 consecutive failures): Network error: Failed to ping master at 127.21.66.126:40885: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
I20260620 23:04:14.985530 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.985424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:43349)
0620 23:04:14.985474 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.985509 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:04:14.985881 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:14.985785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:36565)
0620 23:04:14.985829 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:14.985854 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 23:04:14.985984 4429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:04:14.993570 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4300
I20260620 23:04:14.993824 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:33503
--local_ip_for_outbound_sockets=127.21.66.67
--tserver_master_addrs=127.21.66.126:40885
--webserver_port=33297
--webserver_interface=127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:15.080889 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:15.081127 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:15.081156 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:15.082533 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:15.082582 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:15.082610 4432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:04:15.084118 4432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:33503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=33297
--tserver_master_addrs=127.21.66.126:40885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:15.084360 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:15.084648 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:15.090232 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:15.090328 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:15.090248 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:15.090415 4432 server_base.cc:1034] running on GCE node
I20260620 23:04:15.090835 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:15.091212 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:15.092392 4432 hybrid_clock.cc:648] HybridClock initialized: now 1781996655092366 us; error 38 us; skew 500 ppm
Jun 20 23:04:15 dist-test-slave-qgtk krb5kdc[2994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996655, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:15.096449 4432 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:04:15.097298 4432 webserver.cc:460] Webserver started at http://127.21.66.67:33297/ using document root <none> and password file <none>
I20260620 23:04:15.097590 4432 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:15.097669 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:15.102258 4432 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:04:15.103631 4447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:15.103839 4432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:15.103935 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd"
format_stamp: "Formatted at 2026-06-20 23:04:13 on dist-test-slave-qgtk"
server_key: "05c5b3073fcadf456e616d188329f6ed"
server_key_iv: "c5c2265f58cd538189377c000973f557"
server_key_version: "encryptionkey@0"
I20260620 23:04:15.104460 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:15.115379 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:15.118089 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:15.118304 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:15.118548 4432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:15.118849 4432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:15.118882 4432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:15.118921 4432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:15.118948 4432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:15.125813 4432 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:33503
I20260620 23:04:15.125860 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:33503 every 8 connection(s)
I20260620 23:04:15.126300 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:04:15.127506 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:15.126988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:55199)
0620 23:04:15.127164 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:15.127288 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
W20260620 23:04:15.128024 4561 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:40885 (0 consecutive failures): Network error: Failed to ping master at 127.21.66.126:40885: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
I20260620 23:04:15.128430 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:15.128289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:44045)
0620 23:04:15.128357 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:15.128394 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:04:15.128782 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:15.128684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:50867)
0620 23:04:15.128722 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:15.128751 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 23:04:15.128902 4561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:04:15.129500 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4432
I20260620 23:04:15.895679 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:15.895422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:45055)
0620 23:04:15.895585 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:15.895628 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":59,"threads_started":1}
I20260620 23:04:15.986646 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:15.986350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:58569)
0620 23:04:15.986538 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:15.986587 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260620 23:04:16.129475 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:16.129249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:56985)
0620 23:04:16.129385 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:16.129422 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260620 23:04:16.379696 4568 raft_consensus.cc:482] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:16.379832 4568 raft_consensus.cc:504] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:16.380348 4568 leader_election.cc:288] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503), 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
Jun 20 23:04:16 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.66@KRBTEST.COM
Jun 20 23:04:16 dist-test-slave-qgtk krb5kdc[2994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996654, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.67@KRBTEST.COM
I20260620 23:04:16.392148 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:16.380493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:33503 (local address 127.21.66.65:35835)
0620 23:04:16.380560 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:16.380572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:16.380633 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:16.381708 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:16.381711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:16.382183 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:16.382365 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:16.382372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.383650 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.383651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:16.384627 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:16.384629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.385302 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.385303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:16.385413 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:16.386221 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:16.386226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:16.388366 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:16.390635 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:16.390636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:16.390636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:16.390834 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:16.391109 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:16.391110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:16.391110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:16.391183 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:16.391634 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:16.391635 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:16.391733 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 23:04:16.391842 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:04:16.392037 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":98}
I20260620 23:04:16.392158 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:16.380726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:33561 (local address 127.21.66.66:39931)
0620 23:04:16.380874 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:16.380885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:16.380915 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:04:16.381438 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:16.381440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:16.381495 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:16.381657 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:16.382495 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.383381 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.384745 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.385175 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.385326 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:16.388214 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:16.388220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:16.388220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:16.388239 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:16.390313 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:16.390859 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:16.390863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:16.390863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:16.390966 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:16.391269 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:16.391270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:16.391270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:16.391515 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:16.391611 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:16.391896 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 23:04:16.392036 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":49,"threads_started":1}
I20260620 23:04:16.392211 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:16.380659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:35835 (local address 127.21.66.67:33503)
0620 23:04:16.380853 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:16.380867 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:16.380908 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:04:16.381444 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:16.381447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:16.381500 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:16.381665 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:16.382542 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.383538 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.384752 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.385199 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.385325 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:16.388498 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:16.388503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:16.388503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:16.388514 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:16.390524 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:16.390917 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:16.390921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:16.390921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:16.391015 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:16.391256 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:16.391257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:16.391258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:16.391536 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:16.391634 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:16.392000 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 23:04:16.392110 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260620 23:04:16.392536 4569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:16.380539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:39931 (local address 127.21.66.65:33561)
0620 23:04:16.380726 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:16.380738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:16.380802 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:16.381709 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:16.381711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:16.382092 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:16.382366 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:16.382372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.383670 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.383671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:16.384625 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:16.384627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:16.385293 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:16.385295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:16.385413 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:16.386185 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:16.386196 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:16.387985 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:16.390458 (+ 2473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:16.390468 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:16.390470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:16.390751 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:16.391038 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:16.391039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:16.391039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:16.391185 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:16.391595 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:16.391602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:16.391736 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 23:04:16.391834 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:04:16.392458 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
W20260620 23:04:16.393050 4183 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
W20260620 23:04:16.393263 4182 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348cea95e6314169be2b18059462e979 (127.21.66.66:39931): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
I20260620 23:04:16.393337 4182 leader_election.cc:302] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc65870f1c874e969af2065851e597de; no voters: 348cea95e6314169be2b18059462e979, 7019f5cded4548dfbafcb2d5a2ca6ccd
I20260620 23:04:16.393492 4568 raft_consensus.cc:2726] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:04:16.896389 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:16.896179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:52755)
0620 23:04:16.896305 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:16.896343 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260620 23:04:16.987524 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:16.987247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:56897)
0620 23:04:16.987417 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:16.987475 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260620 23:04:17.130303 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:17.130006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:38491)
0620 23:04:17.130212 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:17.130256 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":64,"threads_started":1}
I20260620 23:04:17.897176 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:17.896918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:39125)
0620 23:04:17.897080 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:17.897130 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260620 23:04:17.988349 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:17.988138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:53303)
0620 23:04:17.988263 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:17.988301 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260620 23:04:18.018374 4577 raft_consensus.cc:482] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:18.018512 4577 raft_consensus.cc:504] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:18.018694 4577 leader_election.cc:288] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503), 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
W20260620 23:04:18.019122 4182 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348cea95e6314169be2b18059462e979 (127.21.66.66:39931): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
W20260620 23:04:18.019220 4183 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
I20260620 23:04:18.019309 4183 leader_election.cc:302] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc65870f1c874e969af2065851e597de; no voters: 348cea95e6314169be2b18059462e979, 7019f5cded4548dfbafcb2d5a2ca6ccd
I20260620 23:04:18.019392 4577 raft_consensus.cc:2726] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:04:18.131135 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:18.130826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:46227)
0620 23:04:18.131038 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:18.131087 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":132,"threads_started":1}
I20260620 23:04:18.897897 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:18.897670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:50605)
0620 23:04:18.897795 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:18.897831 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 23:04:18.989111 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:18.988843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:57921)
0620 23:04:18.989004 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:18.989052 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260620 23:04:19.131853 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:19.131646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:46787)
0620 23:04:19.131767 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:19.131807 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260620 23:04:19.898698 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:19.898483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:42883)
0620 23:04:19.898609 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:19.898652 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260620 23:04:19.989835 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:19.989635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:35453)
0620 23:04:19.989749 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:19.989789 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260620 23:04:20.132618 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:20.132359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:56129)
0620 23:04:20.132514 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:20.132561 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260620 23:04:20.791355 4585 raft_consensus.cc:482] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:20.791488 4585 raft_consensus.cc:504] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:20.791709 4585 leader_election.cc:288] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503), 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
W20260620 23:04:20.792147 4183 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
W20260620 23:04:20.792232 4182 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348cea95e6314169be2b18059462e979 (127.21.66.66:39931): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
I20260620 23:04:20.792311 4182 leader_election.cc:302] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc65870f1c874e969af2065851e597de; no voters: 348cea95e6314169be2b18059462e979, 7019f5cded4548dfbafcb2d5a2ca6ccd
I20260620 23:04:20.792389 4585 raft_consensus.cc:2726] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:04:20.899560 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:20.899315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:50297)
0620 23:04:20.899439 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:20.899510 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260620 23:04:20.990653 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:20.990409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:50087)
0620 23:04:20.990559 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:20.990605 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260620 23:04:21.133471 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:21.133206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:47137)
0620 23:04:21.133340 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:21.133394 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260620 23:04:21.900393 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:21.900101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:51737)
0620 23:04:21.900288 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:21.900337 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260620 23:04:21.991437 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:21.991207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:59283)
0620 23:04:21.991350 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:21.991388 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260620 23:04:22.134341 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:22.134012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:37635)
0620 23:04:22.134236 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:22.134290 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":67,"threads_started":1}
I20260620 23:04:22.901290 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:22.901013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:51911)
0620 23:04:22.901166 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:22.901221 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260620 23:04:22.992285 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:22.991990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:42527)
0620 23:04:22.992148 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:22.992213 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260620 23:04:23.135201 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:23.134913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:56527)
0620 23:04:23.135075 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:23.135128 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260620 23:04:23.902161 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:23.901880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:57943)
0620 23:04:23.902032 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:23.902070 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260620 23:04:23.993106 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:23.992862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:37267)
0620 23:04:23.993009 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:23.993055 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260620 23:04:24.135967 4597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:24.135755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:36897)
0620 23:04:24.135886 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:24.135922 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260620 23:04:24.903049 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:24.902763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:35901)
0620 23:04:24.902947 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:24.902997 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260620 23:04:24.958675 4599 raft_consensus.cc:482] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:24.958786 4599 raft_consensus.cc:504] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7019f5cded4548dfbafcb2d5a2ca6ccd" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 33503 } } peers { permanent_uuid: "348cea95e6314169be2b18059462e979" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 39931 } } peers { permanent_uuid: "fc65870f1c874e969af2065851e597de" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43995 } }
I20260620 23:04:24.959010 4599 leader_election.cc:288] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503), 348cea95e6314169be2b18059462e979 (127.21.66.66:39931)
W20260620 23:04:24.959493 4183 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7019f5cded4548dfbafcb2d5a2ca6ccd (127.21.66.67:33503): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
W20260620 23:04:24.959568 4182 leader_election.cc:341] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 348cea95e6314169be2b18059462e979 (127.21.66.66:39931): Not found: Tablet not found: 29bd071aefb84d3ba055538b749465bf
I20260620 23:04:24.959609 4182 leader_election.cc:302] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc65870f1c874e969af2065851e597de; no voters: 348cea95e6314169be2b18059462e979, 7019f5cded4548dfbafcb2d5a2ca6ccd
I20260620 23:04:24.959736 4599 raft_consensus.cc:2726] T 29bd071aefb84d3ba055538b749465bf P fc65870f1c874e969af2065851e597de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:04:24.994222 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:24.993866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:45599)
0620 23:04:24.994046 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:24.994086 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
I20260620 23:04:25.136853 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:25.136498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:52409)
0620 23:04:25.136734 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:25.136788 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260620 23:04:25.904019 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:25.903781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:35057)
0620 23:04:25.903928 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:25.903973 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260620 23:04:25.995141 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:25.994831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:37137)
0620 23:04:25.994995 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:25.995045 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260620 23:04:26.137763 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:26.137502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:35105)
0620 23:04:26.137664 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:26.137716 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260620 23:04:26.904805 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:26.904558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:40683)
0620 23:04:26.904695 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:26.904758 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260620 23:04:26.995954 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:26.995724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:40125)
0620 23:04:26.995863 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:26.995905 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260620 23:04:27.138684 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:27.138446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:53893)
0620 23:04:27.138588 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:27.138634 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260620 23:04:27.905836 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:27.905574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:33737)
0620 23:04:27.905737 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:27.905786 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260620 23:04:27.996738 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:27.996497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:45093)
0620 23:04:27.996641 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:27.996688 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 23:04:28.139514 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:28.139252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:43879)
0620 23:04:28.139409 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:28.139465 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260620 23:04:28.906703 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:28.906438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:35531)
0620 23:04:28.906611 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:28.906658 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260620 23:04:28.997553 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:28.997323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:53941)
0620 23:04:28.997463 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:28.997504 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260620 23:04:29.140465 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:29.140152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:51659)
0620 23:04:29.140369 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:29.140418 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260620 23:04:29.907647 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:29.907366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.65:49525)
0620 23:04:29.907539 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:29.907591 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260620 23:04:29.998354 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:29.998088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.66:37619)
0620 23:04:29.998269 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:29.998307 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260620 23:04:30.135691 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 4136
I20260620 23:04:30.140726 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 4300
I20260620 23:04:30.141963 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:30.141062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40885 (local address 127.21.66.67:38175)
0620 23:04:30.141866 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:30.141903 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":59,"threads_started":1}
I20260620 23:04:30.145423 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 4432
2026-06-20T23:04:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18482 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:04:30 dist-test-slave-qgtk krb5kdc[4619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:04:30 dist-test-slave-qgtk krb5kdc[4619](info): set up 2 sockets
Jun 20 23:04:30 dist-test-slave-qgtk krb5kdc[4619](info): commencing operation
krb5kdc: starting...
W20260620 23:04:31.206426 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:04:31.337944 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39877
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:31.425495 4632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:31.425733 4632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:31.425760 4632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:31.427044 4632 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:04:31.427099 4632 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:04:31.427115 4632 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:04:31.427129 4632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:31.427145 4632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:31.427160 4632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:31.427173 4632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:31.427187 4632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:31.428685 4632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:31.428900 4632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:31.429086 4632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:31.435510 4638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:31.435508 4640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:31.435612 4632 server_base.cc:1034] running on GCE node
W20260620 23:04:31.435529 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:31.436051 4632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:31.436332 4632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:31.437496 4632 hybrid_clock.cc:648] HybridClock initialized: now 1781996671437477 us; error 31 us; skew 500 ppm
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:31.440268 4632 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:31.440879 4632 webserver.cc:460] Webserver started at http://127.21.66.126:43871/ using document root <none> and password file <none>
I20260620 23:04:31.441082 4632 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:31.441124 4632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:31.441217 4632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:31.442379 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "05dca3c211cd4bb7bf92caad171de45c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "4b5f0846876abff69e1a82393ee676b4"
server_key_iv: "5dc63b070b8aa45f28573ed1e47ea904"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.442703 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "05dca3c211cd4bb7bf92caad171de45c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "4b5f0846876abff69e1a82393ee676b4"
server_key_iv: "5dc63b070b8aa45f28573ed1e47ea904"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.444129 4632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:31.445082 4647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.445351 4632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:31.445560 4632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "05dca3c211cd4bb7bf92caad171de45c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "4b5f0846876abff69e1a82393ee676b4"
server_key_iv: "5dc63b070b8aa45f28573ed1e47ea904"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.445681 4632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:31.465777 4632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:31.468421 4632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:31.468607 4632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:31.473085 4632 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:39877
I20260620 23:04:31.473124 4699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:39877 every 8 connection(s)
I20260620 23:04:31.473521 4632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:31.474570 4700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:31.477000 4700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Bootstrap starting.
I20260620 23:04:31.477653 4700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:31.477980 4700 log.cc:824] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:31.478785 4700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: No bootstrap required, opened a new log
I20260620 23:04:31.480405 4700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:31.480548 4700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:31.480571 4700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dca3c211cd4bb7bf92caad171de45c, State: Initialized, Role: FOLLOWER
I20260620 23:04:31.480751 4700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:31.480823 4700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:31.480844 4700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:31.480926 4700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:31.481470 4700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:31.481590 4700 leader_election.cc:302] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05dca3c211cd4bb7bf92caad171de45c; no voters:
I20260620 23:04:31.481805 4700 leader_election.cc:288] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:04:31.481933 4703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:31.482076 4700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:31.482158 4703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 LEADER]: Becoming Leader. State: Replica: 05dca3c211cd4bb7bf92caad171de45c, State: Running, Role: LEADER
I20260620 23:04:31.482259 4703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:31.482733 4704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05dca3c211cd4bb7bf92caad171de45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } } }
I20260620 23:04:31.482821 4704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:31.483037 4705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05dca3c211cd4bb7bf92caad171de45c. Latest consensus state: current_term: 1 leader_uuid: "05dca3c211cd4bb7bf92caad171de45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } } }
I20260620 23:04:31.483129 4705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:31.483448 4711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:31.483711 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4632
I20260620 23:04:31.483829 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:04:31.483987 4711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:31.484040 21769 external_mini_cluster.cc:1428] Setting key 6175226cad4095dcb430a81314cc5c9e
I20260620 23:04:31.485584 4711 catalog_manager.cc:1347] Generated new cluster ID: 4f0e1e0e13844bb2a33e35336727be01
I20260620 23:04:31.485638 4711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:31.494637 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:31.484715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.21.66.126:39877)
0620 23:04:31.484862 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:31.484874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:31.484907 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:04:31.485033 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:31.485040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:31.485100 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:31.485279 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:31.486061 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.487101 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.487889 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.488355 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.488501 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:31.491139 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:31.491144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:31.491145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:31.491158 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:31.492970 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:31.493362 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:31.493366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:31.493366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:31.493473 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:31.493691 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:31.493692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:31.493692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:31.493950 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:31.494103 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:31.494268 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 23:04:31.494417 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260620 23:04:31.502158 4711 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:04:31.502818 4711 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:31.509341 4711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Generated new TSK 0
I20260620 23:04:31.509608 4711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:04:31.543406 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39877
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:31.635345 4726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:31.635617 4726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:31.635658 4726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:31.636863 4726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:31.636940 4726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:31.636986 4726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:04:31.638561 4726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:31.638842 4726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:31.639092 4726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:31.644773 4734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:31.644706 4732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:31.644727 4731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:31.644994 4726 server_base.cc:1034] running on GCE node
I20260620 23:04:31.645243 4726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:31.645565 4726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:31.646764 4726 hybrid_clock.cc:648] HybridClock initialized: now 1781996671646733 us; error 86 us; skew 500 ppm
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:31.649741 4726 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:04:31.650292 4726 webserver.cc:460] Webserver started at http://127.21.66.65:44275/ using document root <none> and password file <none>
I20260620 23:04:31.650556 4726 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:31.650605 4726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:31.650755 4726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:31.651856 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "5204e8974fde46c7868d56c380da861c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "b6322d4866cfd8e5afc58ab9e0d3459f"
server_key_iv: "c11a49ae51e988ac7568f96ecf1a9998"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.652221 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "5204e8974fde46c7868d56c380da861c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "b6322d4866cfd8e5afc58ab9e0d3459f"
server_key_iv: "c11a49ae51e988ac7568f96ecf1a9998"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.653604 4726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:04:31.654554 4741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.654769 4726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:31.654878 4726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "5204e8974fde46c7868d56c380da861c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "b6322d4866cfd8e5afc58ab9e0d3459f"
server_key_iv: "c11a49ae51e988ac7568f96ecf1a9998"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.654991 4726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:31.667066 4726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:31.669481 4726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:31.669656 4726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:31.669900 4726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:31.670280 4726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:31.670336 4726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.670379 4726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:31.670408 4726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.676551 4726 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:42413
I20260620 23:04:31.676690 4854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:42413 every 8 connection(s)
I20260620 23:04:31.676972 4726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:31.679291 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4726
I20260620 23:04:31.679402 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:04:31.679613 21769 external_mini_cluster.cc:1428] Setting key 9c1807624ce5f2cf85efa093caf96fb5
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:04:31.689035 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:31.677831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:36577 (local address 127.21.66.126:39877)
0620 23:04:31.677892 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:31.677894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:31.678077 (+ 183us) server_negotiation.cc:407] Connection header received
0620 23:04:31.678666 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:31.678668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:31.678684 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:31.678759 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:31.680137 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.680817 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.682364 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.682692 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.682776 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:31.685554 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:31.685556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:31.685556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:31.685566 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:31.687402 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:31.687880 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:31.687881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:31.687881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:31.687942 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:31.688244 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:31.688245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:31.688245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:31.688433 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:31.688516 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:31.688814 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:04:31.688908 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:04:31.689235 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:31.677769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:36577)
0620 23:04:31.677965 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:31.678002 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:31.678542 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:31.678813 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:31.678816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:31.679584 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:31.680022 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:31.680031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.680925 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.680926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:31.682277 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:31.682280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.682864 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.682865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:31.682992 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:31.683560 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:31.683571 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:31.685438 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:31.687516 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:31.687524 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:31.687526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:31.687790 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:31.688044 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:31.688044 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:31.688045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:31.688161 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:31.688524 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:31.688530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:31.688663 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 23:04:31.688765 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:04:31.688920 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260620 23:04:31.690032 4855 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39877
I20260620 23:04:31.690176 4855 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:31.690434 4855 heartbeater.cc:511] Master 127.21.66.126:39877 requested a full tablet report, sending...
I20260620 23:04:31.691028 4664 ts_manager.cc:194] Registered new tserver with Master: 5204e8974fde46c7868d56c380da861c (127.21.66.65:42413)
I20260620 23:04:31.691774 4664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:36577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:04:31.709576 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39877
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:31.792253 4860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:31.792484 4860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:31.792510 4860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:31.793766 4860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:31.793819 4860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:31.793848 4860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:04:31.795415 4860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:31.795708 4860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:31.795964 4860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:31.801633 4865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:31.801612 4868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:31.801635 4866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:31.801613 4860 server_base.cc:1034] running on GCE node
I20260620 23:04:31.802019 4860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:31.802345 4860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:31.803516 4860 hybrid_clock.cc:648] HybridClock initialized: now 1781996671803493 us; error 47 us; skew 500 ppm
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:31.806399 4860 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:04:31.806867 4860 webserver.cc:460] Webserver started at http://127.21.66.66:45315/ using document root <none> and password file <none>
I20260620 23:04:31.807176 4860 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:31.807250 4860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:31.807416 4860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:31.808535 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "788cd9702e8144e9b2162aa59ca97cf5"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "88fea68a4ef0da6782199477e2a9c988"
server_key_iv: "27840f59709e8fea8d0f4f28e3fd20ba"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.808915 4860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "788cd9702e8144e9b2162aa59ca97cf5"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "88fea68a4ef0da6782199477e2a9c988"
server_key_iv: "27840f59709e8fea8d0f4f28e3fd20ba"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.810505 4860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:04:31.811563 4875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.811894 4860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:31.812054 4860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "788cd9702e8144e9b2162aa59ca97cf5"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "88fea68a4ef0da6782199477e2a9c988"
server_key_iv: "27840f59709e8fea8d0f4f28e3fd20ba"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.812160 4860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:31.823604 4860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:31.826601 4860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:31.826805 4860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:31.827078 4860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:31.827440 4860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:31.827515 4860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.827567 4860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:31.827596 4860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.834169 4860 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:38595
I20260620 23:04:31.834219 4988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:38595 every 8 connection(s)
I20260620 23:04:31.834668 4860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:31.844924 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4860
I20260620 23:04:31.845082 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:04:31.845355 21769 external_mini_cluster.cc:1428] Setting key a2d48ca064daf04da833be5dc883e3a2
I20260620 23:04:31.845866 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:31.835530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:58389 (local address 127.21.66.126:39877)
0620 23:04:31.835598 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:31.835600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:31.835880 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:04:31.836395 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:31.836397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:31.836408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:31.836472 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:31.837484 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.838004 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.839241 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.839505 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.839577 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:31.842253 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:31.842254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:31.842254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:31.842262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:31.843920 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:31.844393 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:31.844394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:31.844394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:31.844457 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:31.844909 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:31.844910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:31.844910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:31.845041 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:31.845165 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:31.845653 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 23:04:31.845754 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:04:31.846172 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:31.835519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:58389)
0620 23:04:31.835702 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:31.835758 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:31.836285 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:31.836530 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:31.836534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:31.837162 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:31.837380 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:31.837385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.838100 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.838101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:31.839123 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:31.839126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:31.839602 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:31.839603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:31.839743 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:31.840450 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:31.840459 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:31.842088 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:31.844026 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:31.844032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:31.844034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:31.844278 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:31.844587 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:31.844588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:31.844588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:31.844813 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:31.845194 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:31.845204 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:31.845348 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 23:04:31.845628 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 23:04:31.845854 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260620 23:04:31.846736 4989 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39877
I20260620 23:04:31.846827 4989 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:31.847036 4989 heartbeater.cc:511] Master 127.21.66.126:39877 requested a full tablet report, sending...
I20260620 23:04:31.847422 4664 ts_manager.cc:194] Registered new tserver with Master: 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
I20260620 23:04:31.848052 4664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:58389
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:04:31.875144 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39877
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:31.960963 4994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:31.961217 4994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:31.961261 4994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:31.962543 4994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:31.962610 4994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:31.962656 4994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:04:31.964177 4994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:31.964453 4994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:31.964697 4994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:31.970214 5002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:31.970340 4994 server_base.cc:1034] running on GCE node
W20260620 23:04:31.970229 4999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:31.970474 5000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:31.970705 4994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:31.970983 4994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:31.972147 4994 hybrid_clock.cc:648] HybridClock initialized: now 1781996671972109 us; error 61 us; skew 500 ppm
Jun 20 23:04:31 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:31.974905 4994 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:04:31.975467 4994 webserver.cc:460] Webserver started at http://127.21.66.67:34367/ using document root <none> and password file <none>
I20260620 23:04:31.975703 4994 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:31.975750 4994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:31.975888 4994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:04:31.976986 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "f6605fa98d804ca096581fff7a544a85"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "53b426998b480c43da076c344fe35854"
server_key_iv: "323a542636dfa4f6ff56807130376c15"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.977361 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "f6605fa98d804ca096581fff7a544a85"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "53b426998b480c43da076c344fe35854"
server_key_iv: "323a542636dfa4f6ff56807130376c15"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.978883 4994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:04:31.979843 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.980198 4994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:31.980276 4994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "f6605fa98d804ca096581fff7a544a85"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "53b426998b480c43da076c344fe35854"
server_key_iv: "323a542636dfa4f6ff56807130376c15"
server_key_version: "encryptionkey@0"
I20260620 23:04:31.980377 4994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:31.996024 4994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:31.998473 4994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:31.998641 4994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:31.998912 4994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:31.999269 4994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:31.999305 4994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:31.999331 4994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:31.999344 4994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:32.005404 4994 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:43923
I20260620 23:04:32.005447 5122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:43923 every 8 connection(s)
I20260620 23:04:32.005764 4994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:04:32.010794 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 4994
I20260620 23:04:32.010907 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:04:32.011085 21769 external_mini_cluster.cc:1428] Setting key 799e0cb3a1622669f02d461e65c9727e
Jun 20 23:04:32 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996671, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:32.017462 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.006647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:49425 (local address 127.21.66.126:39877)
0620 23:04:32.006712 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.006714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.006847 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:04:32.007580 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.007581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.007593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.007665 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:32.009012 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.009751 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.011309 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.011608 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.011684 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.014301 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:32.014302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:32.014303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:32.014310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:32.016064 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:32.016482 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:32.016483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:32.016484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:32.016540 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:32.016856 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:32.016857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:32.016857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:32.016976 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:32.017033 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.017314 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.017388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 23:04:32.017604 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.006517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:49425)
0620 23:04:32.006699 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:32.006746 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:32.007400 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:32.007756 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:32.007761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:32.008469 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:32.008884 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.008890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.009863 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.009865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.011110 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.011113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.011700 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.011702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.011860 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.012410 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:32.012419 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:32.014073 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:32.016154 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:32.016163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:32.016165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:32.016411 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:32.016679 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:32.016680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:32.016680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:32.016791 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:32.017039 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:32.017044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:32.017170 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 23:04:32.017254 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:04:32.017384 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260620 23:04:32.018293 5123 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39877
I20260620 23:04:32.018412 5123 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:32.018618 5123 heartbeater.cc:511] Master 127.21.66.126:39877 requested a full tablet report, sending...
I20260620 23:04:32.019134 4664 ts_manager.cc:194] Registered new tserver with Master: f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923)
I20260620 23:04:32.019542 4664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:49425
I20260620 23:04:32.023715 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:04:32.031530 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.025043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51086 (local address 127.21.66.126:39877)
0620 23:04:32.025116 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.025118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.025246 (+ 128us) server_negotiation.cc:407] Connection header received
0620 23:04:32.025291 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.025292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.025301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.025356 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:32.025904 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.026515 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.027331 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.027511 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.027565 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.028916 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:32.028917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:32.028918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:32.028924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:32.030454 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:32.030788 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:32.030789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:32.030789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:32.030840 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:32.031040 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:32.031040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:32.031041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:32.031185 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:32.031260 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.031372 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.031432 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:04:32.033448 4664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:04:32.035538 4664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:04:32.042716 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.038668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.21.66.66:38595)
0620 23:04:32.038815 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.038820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.038979 (+ 159us) server_negotiation.cc:407] Connection header received
0620 23:04:32.039026 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.039029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.039044 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.039194 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:32.039281 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.040037 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.041453 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.042242 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.042320 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.042386 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.042577 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.042628 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260620 23:04:32.042748 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.038393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:38595 (local address 127.0.0.1:50350)
0620 23:04:32.038875 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:32.038883 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:32.038932 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:32.039146 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:32.039147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:32.039148 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:32.039182 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.039183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.040139 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.040140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.041328 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.041330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.042340 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.042341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.042435 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.042452 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 23:04:32.042583 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 23:04:32.042644 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":33,"thread_start_us":89,"threads_started":1}
I20260620 23:04:32.042948 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.038390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42413 (local address 127.0.0.1:53424)
0620 23:04:32.038658 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:32.038702 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:32.038775 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:32.039066 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:32.039070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:32.039072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:32.039140 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.039142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.040179 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.040180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.041490 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.041493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.042497 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.042498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.042589 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.042593 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:04:32.042651 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:04:32.042693 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":24,"thread_start_us":53,"threads_started":1}
I20260620 23:04:32.043601 4923 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa7466d537c4c3f970800294c5b6926 (DEFAULT_TABLE table=test-table [id=fc93c5a99172404dae1fa5182069fc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:32.043972 4923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa7466d537c4c3f970800294c5b6926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:32.044343 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.038410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43923 (local address 127.0.0.1:33972)
0620 23:04:32.038992 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:32.038999 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:32.039034 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:32.040411 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:32.040412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:32.040415 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:32.040453 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.040454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.041838 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.041839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.043207 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.043211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.044111 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.044112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.044183 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.044188 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:32.044254 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:04:32.044288 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":76,"threads_started":1}
I20260620 23:04:32.044418 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.038880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.21.66.67:43923)
0620 23:04:32.040235 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.040239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.040248 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:32.040285 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.040287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.040315 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.040397 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:32.040514 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.041580 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.043288 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.044013 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.044088 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.044154 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.044288 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.044337 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":78,"threads_started":1}
I20260620 23:04:32.045193 5057 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa7466d537c4c3f970800294c5b6926 (DEFAULT_TABLE table=test-table [id=fc93c5a99172404dae1fa5182069fc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:32.045555 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa7466d537c4c3f970800294c5b6926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:32.047396 5140 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Bootstrap starting.
I20260620 23:04:32.047691 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.038505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.21.66.65:42413)
0620 23:04:32.038709 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.038715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.038752 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:04:32.038876 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.038879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.038899 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.038969 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:32.039234 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.040043 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.041610 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.042357 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.042415 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.042461 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.042795 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.042844 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":126,"threads_started":1}
I20260620 23:04:32.048221 5140 tablet_bootstrap.cc:654] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:32.048408 4789 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa7466d537c4c3f970800294c5b6926 (DEFAULT_TABLE table=test-table [id=fc93c5a99172404dae1fa5182069fc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:32.048815 4789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa7466d537c4c3f970800294c5b6926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:32.049016 5141 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Bootstrap starting.
I20260620 23:04:32.048923 5140 log.cc:824] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:32.049666 5141 tablet_bootstrap.cc:654] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:32.049917 5140 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: No bootstrap required, opened a new log
I20260620 23:04:32.049994 5140 ts_tablet_manager.cc:1397] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:04:32.050273 5141 log.cc:824] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:32.051016 5141 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: No bootstrap required, opened a new log
I20260620 23:04:32.051102 5141 ts_tablet_manager.cc:1397] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:32.052374 5144 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Bootstrap starting.
I20260620 23:04:32.052429 5140 raft_consensus.cc:348] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.052673 5140 raft_consensus.cc:374] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:32.052719 5140 raft_consensus.cc:729] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788cd9702e8144e9b2162aa59ca97cf5, State: Initialized, Role: FOLLOWER
I20260620 23:04:32.052836 5140 consensus_queue.cc:260] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.052834 5141 raft_consensus.cc:348] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.053031 5144 tablet_bootstrap.cc:654] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:32.053022 5141 raft_consensus.cc:374] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:32.053071 5141 raft_consensus.cc:729] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6605fa98d804ca096581fff7a544a85, State: Initialized, Role: FOLLOWER
I20260620 23:04:32.053164 5140 ts_tablet_manager.cc:1428] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:04:32.053216 5141 consensus_queue.cc:260] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.053285 4989 heartbeater.cc:503] Master 127.21.66.126:39877 was elected leader, sending a full tablet report...
I20260620 23:04:32.053534 5141 ts_tablet_manager.cc:1428] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:04:32.053568 5123 heartbeater.cc:503] Master 127.21.66.126:39877 was elected leader, sending a full tablet report...
I20260620 23:04:32.053601 5144 log.cc:824] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:32.054559 5144 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: No bootstrap required, opened a new log
I20260620 23:04:32.054651 5144 ts_tablet_manager.cc:1397] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:32.056087 5144 raft_consensus.cc:348] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.056238 5144 raft_consensus.cc:374] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:32.056263 5144 raft_consensus.cc:729] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5204e8974fde46c7868d56c380da861c, State: Initialized, Role: FOLLOWER
I20260620 23:04:32.056351 5144 consensus_queue.cc:260] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.056563 5144 ts_tablet_manager.cc:1428] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:32.056617 4855 heartbeater.cc:503] Master 127.21.66.126:39877 was elected leader, sending a full tablet report...
W20260620 23:04:32.085323 4990 tablet.cc:2369] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:04:32.177759 4856 tablet.cc:2369] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:04:32.187616 5145 raft_consensus.cc:482] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:32.187790 5145 raft_consensus.cc:504] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.188238 5145 leader_election.cc:288] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5204e8974fde46c7868d56c380da861c (127.21.66.65:42413), f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923)
I20260620 23:04:32.192596 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.188567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:54269 (local address 127.21.66.67:43923)
0620 23:04:32.188609 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.188611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.188776 (+ 165us) server_negotiation.cc:407] Connection header received
0620 23:04:32.188818 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.188819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.188828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.188908 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:32.189041 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.189863 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.191479 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.192126 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.192222 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.192231 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.192474 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.192518 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:04:32.192596 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.188515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:40355 (local address 127.21.66.65:42413)
0620 23:04:32.188570 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.188573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.188584 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:04:32.188618 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.188619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.188631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.188710 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:32.188924 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.189847 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.191479 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.192129 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.192215 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.192227 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.192471 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.192517 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:04:32.192600 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.188396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42413 (local address 127.21.66.66:40355)
0620 23:04:32.188456 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:32.188469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:32.188514 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:32.188734 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:32.188735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:32.188738 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:32.188827 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.188829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.189986 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.189988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.191340 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.191344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.192279 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.192280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.192373 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.192374 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 23:04:32.192437 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:04:32.192496 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":42}
I20260620 23:04:32.192606 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.188475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43923 (local address 127.21.66.66:54269)
0620 23:04:32.188665 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:32.188673 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:32.188754 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:32.188910 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:32.188911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:32.188913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:32.188971 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.188972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.189988 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.189988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.191340 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:32.191344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.192280 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.192280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:32.192368 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.192373 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:32.192438 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:04:32.192496 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":17,"thread_start_us":129,"threads_started":1}
I20260620 23:04:32.193069 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa7466d537c4c3f970800294c5b6926" candidate_uuid: "788cd9702e8144e9b2162aa59ca97cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5204e8974fde46c7868d56c380da861c" is_pre_election: true
I20260620 23:04:32.193216 4809 raft_consensus.cc:2454] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 788cd9702e8144e9b2162aa59ca97cf5 in term 0.
I20260620 23:04:32.193238 5077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa7466d537c4c3f970800294c5b6926" candidate_uuid: "788cd9702e8144e9b2162aa59ca97cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6605fa98d804ca096581fff7a544a85" is_pre_election: true
I20260620 23:04:32.193434 5077 raft_consensus.cc:2454] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 788cd9702e8144e9b2162aa59ca97cf5 in term 0.
I20260620 23:04:32.193498 4879 leader_election.cc:302] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5204e8974fde46c7868d56c380da861c, 788cd9702e8144e9b2162aa59ca97cf5; no voters:
I20260620 23:04:32.193660 5145 raft_consensus.cc:2781] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:04:32.193729 5145 raft_consensus.cc:482] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:04:32.193758 5145 raft_consensus.cc:3037] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:32.194581 5145 raft_consensus.cc:504] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.194739 5145 leader_election.cc:288] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [CANDIDATE]: Term 1 election: Requested vote from peers 5204e8974fde46c7868d56c380da861c (127.21.66.65:42413), f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923)
I20260620 23:04:32.195003 5077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa7466d537c4c3f970800294c5b6926" candidate_uuid: "788cd9702e8144e9b2162aa59ca97cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6605fa98d804ca096581fff7a544a85"
I20260620 23:04:32.195101 5077 raft_consensus.cc:3037] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:32.195019 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa7466d537c4c3f970800294c5b6926" candidate_uuid: "788cd9702e8144e9b2162aa59ca97cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5204e8974fde46c7868d56c380da861c"
I20260620 23:04:32.195202 4809 raft_consensus.cc:3037] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:32.196007 5077 raft_consensus.cc:2454] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 788cd9702e8144e9b2162aa59ca97cf5 in term 1.
I20260620 23:04:32.196123 4809 raft_consensus.cc:2454] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 788cd9702e8144e9b2162aa59ca97cf5 in term 1.
I20260620 23:04:32.196306 4878 leader_election.cc:302] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 788cd9702e8144e9b2162aa59ca97cf5, f6605fa98d804ca096581fff7a544a85; no voters:
I20260620 23:04:32.196445 5145 raft_consensus.cc:2781] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:32.196621 5145 raft_consensus.cc:686] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 788cd9702e8144e9b2162aa59ca97cf5, State: Running, Role: LEADER
I20260620 23:04:32.196729 5145 consensus_queue.cc:237] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:32.197621 4664 catalog_manager.cc:5495] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66). New cstate: current_term: 1 leader_uuid: "788cd9702e8144e9b2162aa59ca97cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } health_report { overall_health: UNKNOWN } } }
I20260620 23:04:32.226487 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.223748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.21.66.66:38595)
0620 23:04:32.223831 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:32.223833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:32.223842 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:32.223899 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:32.223900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:32.223920 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:32.223984 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:04:32.224196 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.224704 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.225683 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:32.225883 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:32.225948 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:32.226159 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:04:32.226263 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:04:32.226330 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:32.226386 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 23:04:32.226405 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:04:32.228652 5077 raft_consensus.cc:1261] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Refusing update from remote peer 788cd9702e8144e9b2162aa59ca97cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:32.228644 4809 raft_consensus.cc:1261] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Refusing update from remote peer 788cd9702e8144e9b2162aa59ca97cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:32.229074 5150 consensus_queue.cc:1035] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:32.229256 5145 consensus_queue.cc:1035] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:32.232383 5154 mvcc.cc:205] Tried to move back new op lower bound from 7299058369445916672 to 7299058369318944768. Current Snapshot: MvccSnapshot[applied={T|T < 7299058369445916672}]
I20260620 23:04:32.232425 5155 mvcc.cc:205] Tried to move back new op lower bound from 7299058369445916672 to 7299058369318944768. Current Snapshot: MvccSnapshot[applied={T|T < 7299058369445916672}]
I20260620 23:04:32.234651 4663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:32.234795 4663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:32.235669 4663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Sending DeleteTablet for 3 replicas of tablet 7fa7466d537c4c3f970800294c5b6926
I20260620 23:04:32.236064 5057 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fa7466d537c4c3f970800294c5b6926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:32 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:33972
I20260620 23:04:32.236114 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 4726
I20260620 23:04:32.236115 4923 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fa7466d537c4c3f970800294c5b6926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:32 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:50350
I20260620 23:04:32.236821 5163 tablet_replica.cc:330] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: stopping tablet replica
I20260620 23:04:32.236936 5163 raft_consensus.cc:2229] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:04:32.237035 5162 tablet_replica.cc:330] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: stopping tablet replica
I20260620 23:04:32.237090 5163 raft_consensus.cc:2258] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:32.237241 5162 raft_consensus.cc:2229] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:32.237407 5162 raft_consensus.cc:2258] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:32.237638 5163 ts_tablet_manager.cc:1905] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:32.237977 5162 ts_tablet_manager.cc:1905] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:32.239547 5163 ts_tablet_manager.cc:1918] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:32.239648 5163 log.cc:1196] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/7fa7466d537c4c3f970800294c5b6926
I20260620 23:04:32.239969 5163 ts_tablet_manager.cc:1939] T 7fa7466d537c4c3f970800294c5b6926 P 788cd9702e8144e9b2162aa59ca97cf5: Deleting consensus metadata
I20260620 23:04:32.239975 5162 ts_tablet_manager.cc:1918] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:32.240170 5162 log.cc:1196] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/7fa7466d537c4c3f970800294c5b6926
I20260620 23:04:32.240509 5162 ts_tablet_manager.cc:1939] T 7fa7466d537c4c3f970800294c5b6926 P f6605fa98d804ca096581fff7a544a85: Deleting consensus metadata
I20260620 23:04:32.240691 4649 catalog_manager.cc:4841] TS 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): tablet 7fa7466d537c4c3f970800294c5b6926 (table test-table [id=fc93c5a99172404dae1fa5182069fc93]) successfully deleted
I20260620 23:04:32.240924 4650 catalog_manager.cc:4841] TS f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): tablet 7fa7466d537c4c3f970800294c5b6926 (table test-table [id=fc93c5a99172404dae1fa5182069fc93]) successfully deleted
W20260620 23:04:32.243897 4651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:42413 (error 108)
I20260620 23:04:32.244278 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 4860
I20260620 23:04:32.244480 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:32.244326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42413 (local address 127.0.0.1:53438)
0620 23:04:32.244403 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:32.244440 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 23:04:32.244976 4651 catalog_manager.cc:4584] TS 5204e8974fde46c7868d56c380da861c (127.21.66.65:42413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fa7466d537c4c3f970800294c5b6926: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42413: connect: Connection refused (error 111)
I20260620 23:04:32.248993 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 4994
I20260620 23:04:32.253748 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 4632
I20260620 23:04:52.258894 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39877
--webserver_interface=127.21.66.126
--webserver_port=43871
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:04:52.340938 5165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:52.341156 5165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:52.341179 5165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:52.342381 5165 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:04:52.342433 5165 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:04:52.342447 5165 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:04:52.342458 5165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:04:52.342471 5165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:52.342484 5165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:52.342530 5165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:04:52.342559 5165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:04:52.344097 5165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:52.344378 5165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:52.344635 5165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:52.354156 5174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:52.354194 5172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:52.354226 5165 server_base.cc:1034] running on GCE node
W20260620 23:04:52.354202 5171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:52.354794 5165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:52.355145 5165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:52.356330 5165 hybrid_clock.cc:648] HybridClock initialized: now 1781996692356318 us; error 36 us; skew 500 ppm
Jun 20 23:04:52 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:52.364595 5165 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:04:52.365240 5165 webserver.cc:460] Webserver started at http://127.21.66.126:43871/ using document root <none> and password file <none>
I20260620 23:04:52.365545 5165 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:52.365657 5165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:52.367484 5165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:52.368360 5181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.368592 5165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:52.368721 5165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "05dca3c211cd4bb7bf92caad171de45c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "4b5f0846876abff69e1a82393ee676b4"
server_key_iv: "5dc63b070b8aa45f28573ed1e47ea904"
server_key_version: "encryptionkey@0"
I20260620 23:04:52.369202 5165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:52.385535 5165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:52.388296 5165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:52.388450 5165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:52.393033 5165 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:39877
I20260620 23:04:52.393074 5233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:39877 every 8 connection(s)
I20260620 23:04:52.393537 5165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:04:52.394361 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 5165
I20260620 23:04:52.394789 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:42413
--local_ip_for_outbound_sockets=127.21.66.65
--tserver_master_addrs=127.21.66.126:39877
--webserver_port=44275
--webserver_interface=127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:52.395699 5234 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:04:52.396610 5234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Bootstrap starting.
I20260620 23:04:52.405231 5234 log.cc:824] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:52.407363 5234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:04:52.407707 5234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Bootstrap complete.
I20260620 23:04:52.410082 5234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:52.410418 5234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dca3c211cd4bb7bf92caad171de45c, State: Initialized, Role: FOLLOWER
I20260620 23:04:52.410604 5234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:52.410684 5234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:04:52.410741 5234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:04:52.410792 5234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:04:52.411821 5234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:52.411957 5234 leader_election.cc:302] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05dca3c211cd4bb7bf92caad171de45c; no voters:
I20260620 23:04:52.412161 5234 leader_election.cc:288] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:04:52.412199 5238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:04:52.412314 5238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 LEADER]: Becoming Leader. State: Replica: 05dca3c211cd4bb7bf92caad171de45c, State: Running, Role: LEADER
I20260620 23:04:52.412480 5234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:04:52.412477 5238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:04:52.412889 5239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05dca3c211cd4bb7bf92caad171de45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } } }
I20260620 23:04:52.412962 5239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:52.413309 5240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05dca3c211cd4bb7bf92caad171de45c. Latest consensus state: current_term: 2 leader_uuid: "05dca3c211cd4bb7bf92caad171de45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } } }
I20260620 23:04:52.413401 5243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:04:52.413451 5240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: This master's current role is: LEADER
I20260620 23:04:52.414271 5243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:04:52.414382 5243 catalog_manager.cc:1259] Loaded cluster ID: 4f0e1e0e13844bb2a33e35336727be01
I20260620 23:04:52.414407 5243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:04:52.415542 5243 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:04:52.415721 5243 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Loaded TSK: 0
I20260620 23:04:52.427624 5243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Generated new TSK 1
I20260620 23:04:52.427742 5243 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:04:52.484855 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:52.485049 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:52.485072 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:52.486322 5236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:52.486367 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:52.486395 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:04:52.487982 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:42413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=44275
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:52.488217 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:52.488464 5236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:52.493880 5261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:52.493881 5262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:52.493983 5264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:52.494388 5236 server_base.cc:1034] running on GCE node
I20260620 23:04:52.494612 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:52.494910 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:52.496104 5236 hybrid_clock.cc:648] HybridClock initialized: now 1781996692496088 us; error 30 us; skew 500 ppm
Jun 20 23:04:52 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:52.499264 5236 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:04:52.499926 5236 webserver.cc:460] Webserver started at http://127.21.66.65:44275/ using document root <none> and password file <none>
I20260620 23:04:52.500191 5236 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:52.500283 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:52.501914 5236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:04:52.502934 5271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.503152 5236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:04:52.503270 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "5204e8974fde46c7868d56c380da861c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "b6322d4866cfd8e5afc58ab9e0d3459f"
server_key_iv: "c11a49ae51e988ac7568f96ecf1a9998"
server_key_version: "encryptionkey@0"
I20260620 23:04:52.503777 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:52.512691 5236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:52.515421 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:52.515623 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:52.515897 5236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:52.516389 5278 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:04:52.518345 5236 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:04:52.518482 5236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:04:52.518537 5236 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:04:52.519224 5236 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:04:52.519286 5236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:04:52.519375 5278 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Bootstrap starting.
I20260620 23:04:52.527523 5236 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:42413
I20260620 23:04:52.527705 5385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:42413 every 8 connection(s)
I20260620 23:04:52.528010 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:04:52.530103 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 5236
I20260620 23:04:52.530510 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:38595
--local_ip_for_outbound_sockets=127.21.66.66
--tserver_master_addrs=127.21.66.126:39877
--webserver_port=45315
--webserver_interface=127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:04:52.530505 5278 log.cc:824] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:52.533200 5278 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:04:52.533546 5278 tablet_bootstrap.cc:492] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Bootstrap complete.
I20260620 23:04:52.534799 5278 ts_tablet_manager.cc:1397] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20260620 23:04:52.536584 5278 raft_consensus.cc:348] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:52.536924 5278 raft_consensus.cc:729] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5204e8974fde46c7868d56c380da861c, State: Initialized, Role: FOLLOWER
I20260620 23:04:52.537091 5278 consensus_queue.cc:260] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } }
I20260620 23:04:52.537374 5278 ts_tablet_manager.cc:1428] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Jun 20 23:04:52 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:52.544489 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.528916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:49761)
0620 23:04:52.529311 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.529348 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.529989 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.531066 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.531071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.531517 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:52.531815 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.531822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.533193 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.533194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.534288 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.534291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.535399 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.535401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.535588 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.536256 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:52.536267 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:52.539038 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:52.542038 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:52.542046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:52.542049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:52.542490 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:52.542867 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:52.542868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:52.542869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:52.543030 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:52.543621 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:52.543628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:52.543832 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.543942 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.544159 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":123,"threads_started":1}
I20260620 23:04:52.545732 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.529007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:49761 (local address 127.21.66.126:39877)
0620 23:04:52.529218 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.529228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.529469 (+ 241us) server_negotiation.cc:407] Connection header received
0620 23:04:52.530392 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.530401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.530469 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.530873 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:52.531993 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.533064 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.534402 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.535243 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.535925 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.539193 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:52.539199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:52.539199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:52.539215 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:52.541911 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.542614 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.542619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.542619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.542765 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.543123 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.543124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.543124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.543507 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:52.545174 (+ 1667us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.545268 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.545459 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260620 23:04:52.546880 5386 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39877
I20260620 23:04:52.546991 5386 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:52.547186 5386 heartbeater.cc:511] Master 127.21.66.126:39877 requested a full tablet report, sending...
I20260620 23:04:52.547955 5198 ts_manager.cc:194] Registered new tserver with Master: 5204e8974fde46c7868d56c380da861c (127.21.66.65:42413)
I20260620 23:04:52.549139 5198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:49761
I20260620 23:04:52.550225 5386 heartbeater.cc:503] Master 127.21.66.126:39877 was elected leader, sending a full tablet report...
I20260620 23:04:52.554803 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.549412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.21.66.65:42413)
0620 23:04:52.550582 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.550587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.550598 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:04:52.550635 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.550637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.550652 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.550785 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:52.550947 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.551998 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.553697 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.554383 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.554458 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.554511 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.554700 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.554738 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260620 23:04:52.554821 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.548589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42413 (local address 127.0.0.1:35534)
0620 23:04:52.550220 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.550260 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.550408 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.550795 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.550798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.550800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:52.550869 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.550871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.552083 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.552084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.553582 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.553586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.554473 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.554474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.554575 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.554591 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.554674 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.554732 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"thread_start_us":95,"threads_started":1}
I20260620 23:04:52.555390 5320 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fa7466d537c4c3f970800294c5b6926 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:32 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:35534
I20260620 23:04:52.556035 5395 tablet_replica.cc:330] stopping tablet replica
I20260620 23:04:52.556149 5395 raft_consensus.cc:2229] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:52.556216 5395 raft_consensus.cc:2258] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:52.556564 5395 ts_tablet_manager.cc:1905] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:52.558692 5395 ts_tablet_manager.cc:1918] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:52.558755 5395 log.cc:1196] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal/wals/7fa7466d537c4c3f970800294c5b6926
I20260620 23:04:52.559096 5395 ts_tablet_manager.cc:1939] T 7fa7466d537c4c3f970800294c5b6926 P 5204e8974fde46c7868d56c380da861c: Deleting consensus metadata
I20260620 23:04:52.559744 5185 catalog_manager.cc:4841] TS 5204e8974fde46c7868d56c380da861c (127.21.66.65:42413): tablet 7fa7466d537c4c3f970800294c5b6926 (table test-table [id=fc93c5a99172404dae1fa5182069fc93]) successfully deleted
W20260620 23:04:52.631603 5391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:52.631877 5391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:52.631922 5391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:52.633229 5391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:52.633298 5391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:52.633351 5391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:04:52.634897 5391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:38595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=45315
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:52.635159 5391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:52.635416 5391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:52.640739 5401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:52.640745 5403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:52.640745 5400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:52.640763 5391 server_base.cc:1034] running on GCE node
I20260620 23:04:52.641232 5391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:52.641556 5391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:52.642724 5391 hybrid_clock.cc:648] HybridClock initialized: now 1781996692642714 us; error 48 us; skew 500 ppm
Jun 20 23:04:52 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:52.645287 5391 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:04:52.645977 5391 webserver.cc:460] Webserver started at http://127.21.66.66:45315/ using document root <none> and password file <none>
I20260620 23:04:52.646257 5391 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:52.646335 5391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:52.647918 5391 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:52.648623 5410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.648929 5391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:52.649037 5391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "788cd9702e8144e9b2162aa59ca97cf5"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "88fea68a4ef0da6782199477e2a9c988"
server_key_iv: "27840f59709e8fea8d0f4f28e3fd20ba"
server_key_version: "encryptionkey@0"
I20260620 23:04:52.649430 5391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:52.658385 5391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:52.660667 5391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:52.660792 5391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:52.661015 5391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:52.661273 5391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:52.661303 5391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.661327 5391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:52.661341 5391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.667243 5391 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:38595
I20260620 23:04:52.667440 5523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:38595 every 8 connection(s)
I20260620 23:04:52.667649 5391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:04:52.669355 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 5391
I20260620 23:04:52.669746 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:43923
--local_ip_for_outbound_sockets=127.21.66.67
--tserver_master_addrs=127.21.66.126:39877
--webserver_port=34367
--webserver_interface=127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:04:52 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:52.678908 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.668507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:50965 (local address 127.21.66.126:39877)
0620 23:04:52.668578 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.668580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.668771 (+ 191us) server_negotiation.cc:407] Connection header received
0620 23:04:52.669444 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.669445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.669467 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.669551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:52.671097 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.671628 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.672782 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.673014 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.673095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.675799 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:52.675801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:52.675801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:52.675809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:52.677365 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.677815 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.677817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.677817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.677887 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.678200 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.678201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.678202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.678327 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:52.678411 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.678703 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.678800 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:04:52.679126 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.668500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:50965)
0620 23:04:52.668666 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.668703 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.669227 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.669680 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.669685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.670664 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:52.670978 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.670986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.671718 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.671719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.672642 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.672644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.673101 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.673102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.673236 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.673804 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:52.673812 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:52.675686 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:52.677473 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:52.677482 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:52.677484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:52.677720 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:52.677958 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:52.677959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:52.677959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:52.678086 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:52.678403 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:52.678410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:52.678563 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.678670 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.678854 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260620 23:04:52.679991 5524 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39877
I20260620 23:04:52.680099 5524 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:52.680346 5524 heartbeater.cc:511] Master 127.21.66.126:39877 requested a full tablet report, sending...
I20260620 23:04:52.680842 5198 ts_manager.cc:194] Registered new tserver with Master: 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
I20260620 23:04:52.681354 5198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:50965
W20260620 23:04:52.757472 5527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:04:52.757716 5527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:04:52.757755 5527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:04:52.759016 5527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:04:52.759085 5527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:04:52.759131 5527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:04:52.760644 5527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:43923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=34367
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:04:52.760911 5527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:04:52.761152 5527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:04:52.766454 5532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:52.766507 5535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:04:52.766450 5533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:04:52.766785 5527 server_base.cc:1034] running on GCE node
I20260620 23:04:52.766994 5527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:04:52.767221 5527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:04:52.768438 5527 hybrid_clock.cc:648] HybridClock initialized: now 1781996692768429 us; error 32 us; skew 500 ppm
Jun 20 23:04:52 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:04:52.771287 5527 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:04:52.771903 5527 webserver.cc:460] Webserver started at http://127.21.66.67:34367/ using document root <none> and password file <none>
I20260620 23:04:52.772154 5527 fs_manager.cc:359] Metadata directory not provided
I20260620 23:04:52.772243 5527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:04:52.773722 5527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:04:52.774523 5542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.774848 5527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:04:52.774963 5527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "f6605fa98d804ca096581fff7a544a85"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "53b426998b480c43da076c344fe35854"
server_key_iv: "323a542636dfa4f6ff56807130376c15"
server_key_version: "encryptionkey@0"
I20260620 23:04:52.775436 5527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:04:52.791546 5527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:04:52.794071 5527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:04:52.794278 5527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:04:52.794543 5527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:04:52.794863 5527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:04:52.794920 5527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.794963 5527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:04:52.794993 5527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.800920 5527 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:43923
I20260620 23:04:52.800962 5655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:43923 every 8 connection(s)
I20260620 23:04:52.801322 5527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:04:52.805786 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 5527
Jun 20 23:04:52 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996692, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:04:52.812079 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.802070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:48971 (local address 127.21.66.126:39877)
0620 23:04:52.802204 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.802206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.802394 (+ 188us) server_negotiation.cc:407] Connection header received
0620 23:04:52.802874 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.802876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.802889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.802962 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:52.803907 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.804477 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.805477 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.805715 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.805816 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.808526 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:52.808527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:52.808527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:52.808534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:52.810377 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.810886 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.810887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.810888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.810950 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.811282 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.811283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.811283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.811427 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:52.811520 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.811876 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.811978 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 23:04:52.812249 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.802061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:48971)
0620 23:04:52.802240 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.802282 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.802770 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.803018 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.803022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.803594 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:04:52.803795 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.803801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.804572 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.804573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.805381 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.805383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.805836 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.805837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.805997 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.806744 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:04:52.806753 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:04:52.808432 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:04:52.810476 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:52.810482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:52.810484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:52.810782 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:52.811071 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:04:52.811071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:04:52.811072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:04:52.811195 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:04:52.811510 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:04:52.811517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:04:52.811681 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.811802 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.811988 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260620 23:04:52.813122 5656 heartbeater.cc:347] Connected to a master server at 127.21.66.126:39877
I20260620 23:04:52.813261 5656 heartbeater.cc:464] Registering TS with master...
I20260620 23:04:52.813544 5656 heartbeater.cc:511] Master 127.21.66.126:39877 requested a full tablet report, sending...
I20260620 23:04:52.814060 5198 ts_manager.cc:194] Registered new tserver with Master: f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923)
I20260620 23:04:52.814551 5198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:48971
I20260620 23:04:52.814687 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.806239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.21.66.126:39877)
0620 23:04:52.806439 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.806441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.806599 (+ 158us) server_negotiation.cc:407] Connection header received
0620 23:04:52.806684 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.806685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.806698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.806783 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:04:52.807500 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.808085 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.809039 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.809248 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.809301 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.810712 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:04:52.810713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:04:52.810714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:04:52.810721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:04:52.812431 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.812948 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.812949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.812950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.813024 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:04:52.813299 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:04:52.813305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:04:52.813306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:04:52.813502 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:04:52.813725 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.813766 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.813863 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260620 23:04:52.815428 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:04:52.818305 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.815815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.21.66.126:39877)
0620 23:04:52.815858 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.815862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.815987 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:04:52.816046 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.816047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.816077 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.816141 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:04:52.816223 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.816726 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.817518 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.817735 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.817804 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.817976 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:04:52.818071 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:04:52.818204 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.818226 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.818246 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:04:52.818593 5198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:04:52.820258 5198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:04:52.822582 5320 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 (DEFAULT_TABLE table=test-table [id=15c106a05cdb4db58742a3c55ed7142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:52.822816 5320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:52.824270 5278 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Bootstrap starting.
I20260620 23:04:52.824653 5278 tablet_bootstrap.cc:654] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:52.825328 5278 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: No bootstrap required, opened a new log
I20260620 23:04:52.825379 5278 ts_tablet_manager.cc:1397] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 23:04:52.825479 5278 raft_consensus.cc:348] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.825553 5278 raft_consensus.cc:374] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:52.825609 5278 raft_consensus.cc:729] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5204e8974fde46c7868d56c380da861c, State: Initialized, Role: FOLLOWER
I20260620 23:04:52.825649 5278 consensus_queue.cc:260] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.825734 5278 ts_tablet_manager.cc:1428] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 23:04:52.826648 5662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.822350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:38595 (local address 127.0.0.1:57624)
0620 23:04:52.822546 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.822556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.822604 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.823125 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.823126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.823128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:52.823203 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.823205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.824220 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.824222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.825333 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.825336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.826401 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.826402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.826466 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.826471 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.826529 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.826572 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":11,"thread_start_us":129,"threads_started":1}
I20260620 23:04:52.826679 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.822692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.21.66.66:38595)
0620 23:04:52.822936 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.822941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.822952 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:04:52.822994 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.822996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.823013 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.823102 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:52.823289 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.824121 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.825489 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.826301 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.826397 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.826461 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.826543 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.826581 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":142,"threads_started":1}
I20260620 23:04:52.827744 5458 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 (DEFAULT_TABLE table=test-table [id=15c106a05cdb4db58742a3c55ed7142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:52.828115 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.822248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43923 (local address 127.0.0.1:52982)
0620 23:04:52.822285 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.822295 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.822328 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.824734 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.824735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.824737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:52.824795 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.824797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.826098 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.826099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.827011 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.827014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.827900 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.827901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.827979 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.827982 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.828036 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.828071 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 23:04:52.828132 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:52.828203 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.822431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.21.66.67:43923)
0620 23:04:52.824549 (+ 2118us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.824554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.824565 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:04:52.824603 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.824605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.824622 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.824875 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:52.824965 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.826022 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.827095 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.827806 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.827882 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.827937 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.828076 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.828119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1975,"thread_start_us":105,"threads_started":1}
I20260620 23:04:52.828754 5590 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 (DEFAULT_TABLE table=test-table [id=15c106a05cdb4db58742a3c55ed7142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:04:52.829062 5590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:04:52.831153 5666 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Bootstrap starting.
I20260620 23:04:52.831864 5667 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Bootstrap starting.
I20260620 23:04:52.831911 5666 tablet_bootstrap.cc:654] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:52.832545 5666 log.cc:824] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:52.832592 5667 tablet_bootstrap.cc:654] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Neither blocks nor log segments found. Creating new log.
I20260620 23:04:52.833164 5667 log.cc:824] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Log is configured to *not* fsync() on all Append() calls
I20260620 23:04:52.833361 5666 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: No bootstrap required, opened a new log
I20260620 23:04:52.833436 5666 ts_tablet_manager.cc:1397] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:52.833958 5667 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: No bootstrap required, opened a new log
I20260620 23:04:52.834028 5667 ts_tablet_manager.cc:1397] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:04:52.835484 5666 raft_consensus.cc:348] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.835671 5666 raft_consensus.cc:374] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:52.835721 5666 raft_consensus.cc:729] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788cd9702e8144e9b2162aa59ca97cf5, State: Initialized, Role: FOLLOWER
I20260620 23:04:52.835860 5666 consensus_queue.cc:260] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.836139 5666 ts_tablet_manager.cc:1428] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:04:52.836048 5667 raft_consensus.cc:348] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.836169 5524 heartbeater.cc:503] Master 127.21.66.126:39877 was elected leader, sending a full tablet report...
I20260620 23:04:52.836225 5667 raft_consensus.cc:374] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:04:52.836283 5667 raft_consensus.cc:729] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6605fa98d804ca096581fff7a544a85, State: Initialized, Role: FOLLOWER
I20260620 23:04:52.836400 5667 consensus_queue.cc:260] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.836687 5667 ts_tablet_manager.cc:1428] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:04:52.836719 5656 heartbeater.cc:503] Master 127.21.66.126:39877 was elected leader, sending a full tablet report...
I20260620 23:04:52.842655 5392 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:04:52.842866 5392 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.843235 5392 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
I20260620 23:04:52.847724 5664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.843446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:40513 (local address 127.21.66.66:38595)
0620 23:04:52.843490 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.843493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.843502 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:04:52.843593 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.843595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.843606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.843665 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:52.844169 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.845043 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.846675 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.847296 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.847365 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.847375 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.847588 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.847657 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:04:52.847746 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.843648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:42287 (local address 127.21.66.67:43923)
0620 23:04:52.843698 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.843700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.843823 (+ 123us) server_negotiation.cc:407] Connection header received
0620 23:04:52.843878 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.843879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.843886 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.843930 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:04:52.844105 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.844982 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.846675 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.847278 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.847447 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.847454 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.847636 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.847675 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:04:52.847723 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.843398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:38595 (local address 127.21.66.65:40513)
0620 23:04:52.843456 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.843468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.843508 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.843674 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.843675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.843677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:52.844082 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.844084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.845142 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.845142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.846571 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.846574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.847377 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.847379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.847494 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.847499 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.847566 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.847631 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":3}
I20260620 23:04:52.847728 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.843437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43923 (local address 127.21.66.65:42287)
0620 23:04:52.843728 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.843739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:04:52.843819 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:04:52.843936 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:04:52.843937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:04:52.843938 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:04:52.844012 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.844014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.845121 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.845123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.846574 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:04:52.846575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.847485 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.847486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:04:52.847559 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.847564 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:04:52.847622 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 23:04:52.847652 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":17,"thread_start_us":123,"threads_started":1}
I20260620 23:04:52.848158 5478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc302f90d1a4e9b84420eb1c74eb2a5" candidate_uuid: "5204e8974fde46c7868d56c380da861c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788cd9702e8144e9b2162aa59ca97cf5" is_pre_election: true
I20260620 23:04:52.848173 5610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc302f90d1a4e9b84420eb1c74eb2a5" candidate_uuid: "5204e8974fde46c7868d56c380da861c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6605fa98d804ca096581fff7a544a85" is_pre_election: true
I20260620 23:04:52.848325 5478 raft_consensus.cc:2454] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5204e8974fde46c7868d56c380da861c in term 0.
I20260620 23:04:52.848325 5610 raft_consensus.cc:2454] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5204e8974fde46c7868d56c380da861c in term 0.
I20260620 23:04:52.848598 5274 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5204e8974fde46c7868d56c380da861c, f6605fa98d804ca096581fff7a544a85; no voters:
I20260620 23:04:52.848749 5392 raft_consensus.cc:2781] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:04:52.848801 5392 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:04:52.848874 5392 raft_consensus.cc:3037] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:52.849606 5392 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.849759 5392 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 1 election: Requested vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
I20260620 23:04:52.849921 5478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc302f90d1a4e9b84420eb1c74eb2a5" candidate_uuid: "5204e8974fde46c7868d56c380da861c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788cd9702e8144e9b2162aa59ca97cf5"
I20260620 23:04:52.850008 5478 raft_consensus.cc:3037] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:52.850015 5610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc302f90d1a4e9b84420eb1c74eb2a5" candidate_uuid: "5204e8974fde46c7868d56c380da861c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6605fa98d804ca096581fff7a544a85"
I20260620 23:04:52.850104 5610 raft_consensus.cc:3037] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:04:52.850781 5478 raft_consensus.cc:2454] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5204e8974fde46c7868d56c380da861c in term 1.
I20260620 23:04:52.850940 5610 raft_consensus.cc:2454] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5204e8974fde46c7868d56c380da861c in term 1.
I20260620 23:04:52.850966 5273 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5204e8974fde46c7868d56c380da861c, 788cd9702e8144e9b2162aa59ca97cf5; no voters:
I20260620 23:04:52.851070 5392 raft_consensus.cc:2781] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:04:52.851248 5392 raft_consensus.cc:686] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 LEADER]: Becoming Leader. State: Replica: 5204e8974fde46c7868d56c380da861c, State: Running, Role: LEADER
I20260620 23:04:52.851307 5392 consensus_queue.cc:237] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:04:52.852133 5197 catalog_manager.cc:5495] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5204e8974fde46c7868d56c380da861c (127.21.66.65). New cstate: current_term: 1 leader_uuid: "5204e8974fde46c7868d56c380da861c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } health_report { overall_health: HEALTHY } } }
I20260620 23:04:52.861096 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.858695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.21.66.65:42413)
0620 23:04:52.858757 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:04:52.858759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:04:52.858771 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:04:52.858809 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:04:52.858810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:04:52.858828 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:04:52.858887 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:04:52.859017 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.859481 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.860237 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:04:52.860464 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:04:52.860522 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:04:52.860750 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:04:52.860856 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:04:52.860920 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:04:52.861012 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:04:52.861034 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:04:52.863117 5478 raft_consensus.cc:1261] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 FOLLOWER]: Refusing update from remote peer 5204e8974fde46c7868d56c380da861c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:52.863129 5610 raft_consensus.cc:1261] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Refusing update from remote peer 5204e8974fde46c7868d56c380da861c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:04:52.863566 5673 consensus_queue.cc:1035] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:52.863710 5392 consensus_queue.cc:1035] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [LEADER]: Connected to new peer: Peer: permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:04:52.866631 5676 mvcc.cc:205] Tried to move back new op lower bound from 7299058453964197888 to 7299058453919793152. Current Snapshot: MvccSnapshot[applied={T|T < 7299058453964197888}]
I20260620 23:04:52.867501 5678 mvcc.cc:205] Tried to move back new op lower bound from 7299058453964197888 to 7299058453919793152. Current Snapshot: MvccSnapshot[applied={T|T < 7299058453964197888}]
I20260620 23:04:52.867631 5677 mvcc.cc:205] Tried to move back new op lower bound from 7299058453964197888 to 7299058453919793152. Current Snapshot: MvccSnapshot[applied={T|T < 7299058453964197888}]
I20260620 23:04:52.868790 5197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:52.868891 5197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:04:52.869688 5197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Sending DeleteTablet for 3 replicas of tablet 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:04:52.870077 5320 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:52 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:35534
I20260620 23:04:52.870108 5590 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:52 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52982
I20260620 23:04:52.870231 5395 tablet_replica.cc:330] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: stopping tablet replica
I20260620 23:04:52.870293 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 5236
I20260620 23:04:52.870312 5395 raft_consensus.cc:2229] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:04:52.870401 5395 raft_consensus.cc:2258] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:52.870477 5458 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:04:52 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:57624
I20260620 23:04:52.871048 5686 tablet_replica.cc:330] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: stopping tablet replica
I20260620 23:04:52.871166 5686 raft_consensus.cc:2229] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:52.871291 5686 raft_consensus.cc:2258] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:52.871726 5686 ts_tablet_manager.cc:1905] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:52.873876 5686 ts_tablet_manager.cc:1918] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:52.873961 5686 log.cc:1196] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:04:52.874264 5686 ts_tablet_manager.cc:1939] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 788cd9702e8144e9b2162aa59ca97cf5: Deleting consensus metadata
I20260620 23:04:52.874282 5685 tablet_replica.cc:330] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: stopping tablet replica
I20260620 23:04:52.874414 5685 raft_consensus.cc:2229] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:04:52.874544 5685 raft_consensus.cc:2258] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:04:52.874712 5183 catalog_manager.cc:4841] TS 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 (table test-table [id=15c106a05cdb4db58742a3c55ed7142e]) successfully deleted
I20260620 23:04:52.875116 5685 ts_tablet_manager.cc:1905] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:04:52.876492 5685 ts_tablet_manager.cc:1918] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:04:52.876554 5685 log.cc:1196] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:04:52.876893 5685 ts_tablet_manager.cc:1939] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P f6605fa98d804ca096581fff7a544a85: Deleting consensus metadata
I20260620 23:04:52.877367 5184 catalog_manager.cc:4841] TS f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): tablet 6fc302f90d1a4e9b84420eb1c74eb2a5 (table test-table [id=15c106a05cdb4db58742a3c55ed7142e]) successfully deleted
W20260620 23:04:52.877507 5185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:42413 (error 108)
I20260620 23:04:52.877789 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 5391
I20260620 23:04:52.878427 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:04:52.877923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42413 (local address 127.0.0.1:35548)
0620 23:04:52.878336 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 23:04:52.878381 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
W20260620 23:04:52.880203 5185 catalog_manager.cc:4584] TS 5204e8974fde46c7868d56c380da861c (127.21.66.65:42413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fc302f90d1a4e9b84420eb1c74eb2a5: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42413: connect: Connection refused (error 111)
I20260620 23:04:52.884584 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 5527
I20260620 23:04:52.889590 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 5165
I20260620 23:05:12.895550 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39877
--webserver_interface=127.21.66.126
--webserver_port=43871
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:12.978613 5688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:12.978828 5688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:12.978853 5688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:12.980046 5688 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:05:12.980089 5688 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:05:12.980103 5688 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:05:12.980114 5688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:12.980127 5688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:12.980140 5688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:12.980154 5688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:12.980208 5688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:12.981722 5688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:12.981981 5688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:12.982260 5688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:12.988173 5693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:12.988198 5688 server_base.cc:1034] running on GCE node
W20260620 23:05:12.988240 5694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:12.988178 5696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:12.988646 5688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:12.988966 5688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:12.990211 5688 hybrid_clock.cc:648] HybridClock initialized: now 1781996712990178 us; error 48 us; skew 500 ppm
Jun 20 23:05:12 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996712, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:12.997447 5688 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:12.998013 5688 webserver.cc:460] Webserver started at http://127.21.66.126:43871/ using document root <none> and password file <none>
I20260620 23:05:12.998306 5688 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:12.998387 5688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:12.999780 5688 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:13.000579 5703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.000813 5688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:13.000897 5688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "05dca3c211cd4bb7bf92caad171de45c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "4b5f0846876abff69e1a82393ee676b4"
server_key_iv: "5dc63b070b8aa45f28573ed1e47ea904"
server_key_version: "encryptionkey@0"
I20260620 23:05:13.001246 5688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:13.026325 5688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:13.028643 5688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:13.028767 5688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:13.033123 5688 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:39877
I20260620 23:05:13.033195 5755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:39877 every 8 connection(s)
I20260620 23:05:13.033602 5688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:13.035216 5756 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:05:13.035920 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Bootstrap starting.
I20260620 23:05:13.041281 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 5688
I20260620 23:05:13.041600 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:42413
--local_ip_for_outbound_sockets=127.21.66.65
--tserver_master_addrs=127.21.66.126:39877
--webserver_port=44275
--webserver_interface=127.21.66.65
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:05:13.044432 5756 log.cc:824] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:13.047142 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:05:13.047467 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Bootstrap complete.
I20260620 23:05:13.049705 5756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:05:13.049962 5756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dca3c211cd4bb7bf92caad171de45c, State: Initialized, Role: FOLLOWER
I20260620 23:05:13.050187 5756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:05:13.050411 5756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:13.050518 5756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:13.050575 5756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:05:13.051637 5756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:05:13.051769 5756 leader_election.cc:302] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05dca3c211cd4bb7bf92caad171de45c; no voters:
I20260620 23:05:13.051980 5756 leader_election.cc:288] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:05:13.052019 5760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:05:13.052147 5760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [term 3 LEADER]: Becoming Leader. State: Replica: 05dca3c211cd4bb7bf92caad171de45c, State: Running, Role: LEADER
I20260620 23:05:13.052259 5756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:13.052232 5760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } }
I20260620 23:05:13.052572 5761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "05dca3c211cd4bb7bf92caad171de45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } } }
I20260620 23:05:13.052665 5761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:13.052572 5760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05dca3c211cd4bb7bf92caad171de45c. Latest consensus state: current_term: 3 leader_uuid: "05dca3c211cd4bb7bf92caad171de45c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dca3c211cd4bb7bf92caad171de45c" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39877 } } }
I20260620 23:05:13.052977 5760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:13.053231 5765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:13.054039 5765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:13.054215 5765 catalog_manager.cc:1259] Loaded cluster ID: 4f0e1e0e13844bb2a33e35336727be01
I20260620 23:05:13.054409 5765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:13.055613 5765 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:13.055766 5765 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Loaded TSK: 0
I20260620 23:05:13.055799 5765 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05dca3c211cd4bb7bf92caad171de45c: Loaded TSK: 1
F20260620 23:05:13.056036 5765 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc550c399ad google::LogMessage::Fail() at ??:0
@ 0x7fc550c3c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc550c394de google::LogMessage::Flush() at ??:0
@ 0x7fc550c3aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5533a6104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc5533a3d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc558b931cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc558b9348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc558b93594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc558b59c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc558b63ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc552b070d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc552afc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc5539606db start_thread at ??:0
@ 0x7fc54f73761f clone at ??:0
W20260620 23:05:13.156167 5758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:13.156562 5758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:13.156606 5758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:13.159013 5758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:13.159106 5758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:13.159154 5758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:05:13.161612 5758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:42413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=44275
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:13.161962 5758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:13.162281 5758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:13.168659 5798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:13.168661 5800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:13.169051 5797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:13.169695 5758 server_base.cc:1034] running on GCE node
I20260620 23:05:13.169939 5758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:13.170243 5758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:13.171384 5758 hybrid_clock.cc:648] HybridClock initialized: now 1781996713171352 us; error 44 us; skew 500 ppm
Jun 20 23:05:13 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996713, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:13.176093 5758 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:05:13.176781 5758 webserver.cc:460] Webserver started at http://127.21.66.65:44275/ using document root <none> and password file <none>
I20260620 23:05:13.177083 5758 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:13.177157 5758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:13.179283 5758 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:13.180227 5807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.180478 5758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:13.180585 5758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "5204e8974fde46c7868d56c380da861c"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "b6322d4866cfd8e5afc58ab9e0d3459f"
server_key_iv: "c11a49ae51e988ac7568f96ecf1a9998"
server_key_version: "encryptionkey@0"
I20260620 23:05:13.181120 5758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:13.192071 5758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:13.195123 5758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:13.195288 5758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:13.195551 5758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:13.196017 5814 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 23:05:13.198287 5758 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:05:13.198366 5758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 23:05:13.198419 5758 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:05:13.199137 5758 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:05:13.199198 5758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 23:05:13.199282 5814 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Bootstrap starting.
I20260620 23:05:13.206606 5758 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:42413
I20260620 23:05:13.206759 5921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:42413 every 8 connection(s)
I20260620 23:05:13.207047 5758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:13.209055 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 5758
I20260620 23:05:13.209474 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:38595
--local_ip_for_outbound_sockets=127.21.66.66
--tserver_master_addrs=127.21.66.126:39877
--webserver_port=45315
--webserver_interface=127.21.66.66
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:05:13.210433 5814 log.cc:824] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:13.213438 5814 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:05:13.213783 5814 tablet_bootstrap.cc:492] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Bootstrap complete.
I20260620 23:05:13.215060 5814 ts_tablet_manager.cc:1397] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260620 23:05:13.216602 5814 raft_consensus.cc:348] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:13.216950 5814 raft_consensus.cc:729] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5204e8974fde46c7868d56c380da861c, State: Initialized, Role: FOLLOWER
I20260620 23:05:13.217120 5814 consensus_queue.cc:260] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:13.217432 5814 ts_tablet_manager.cc:1428] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260620 23:05:13.288060 5924 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:05:13.207841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:34687)
0620 23:05:13.208011 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.208053 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:13.208514 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:13.287885 (+ 79371us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
W20260620 23:05:13.288316 5922 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:39877 (0 consecutive failures): Network error: Failed to ping master at 127.21.66.126:39877: Client connection negotiation failed: client connection to 127.21.66.126:39877: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:05:13.288689 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.288563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:60353)
0620 23:05:13.288621 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.288668 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 23:05:13.289090 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.288997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:57465)
0620 23:05:13.289048 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.289073 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:05:13.289227 5922 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:05:13.307528 5926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:13.307771 5926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:13.307811 5926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:13.309029 5926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:13.309105 5926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:13.309154 5926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:05:13.310945 5926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:38595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=45315
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:13.311260 5926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:13.311507 5926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:13.317233 5935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:13.317251 5933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:13.317298 5932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:13.317274 5926 server_base.cc:1034] running on GCE node
I20260620 23:05:13.317813 5926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:13.318190 5926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:13.319389 5926 hybrid_clock.cc:648] HybridClock initialized: now 1781996713319382 us; error 39 us; skew 500 ppm
Jun 20 23:05:13 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996713, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:13.322274 5926 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:05:13.322736 5926 webserver.cc:460] Webserver started at http://127.21.66.66:45315/ using document root <none> and password file <none>
I20260620 23:05:13.322997 5926 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:13.323071 5926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:13.324419 5926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:13.325179 5942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.325364 5926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:05:13.325459 5926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "788cd9702e8144e9b2162aa59ca97cf5"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "88fea68a4ef0da6782199477e2a9c988"
server_key_iv: "27840f59709e8fea8d0f4f28e3fd20ba"
server_key_version: "encryptionkey@0"
I20260620 23:05:13.325822 5926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:13.335098 5926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:13.337410 5926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:13.337641 5926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:13.337939 5926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:13.338341 5926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:13.338410 5926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.338457 5926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:13.338500 5926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.345319 5926 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:38595
I20260620 23:05:13.345366 6055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:38595 every 8 connection(s)
I20260620 23:05:13.345755 5926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:13.346956 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.346540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:42115)
0620 23:05:13.346725 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.346803 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
W20260620 23:05:13.347465 6056 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:39877 (0 consecutive failures): Network error: Failed to ping master at 127.21.66.126:39877: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
I20260620 23:05:13.347812 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.347718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:44819)
0620 23:05:13.347761 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.347792 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 23:05:13.348161 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 5926
I20260620 23:05:13.348165 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.348078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:49059)
0620 23:05:13.348126 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.348149 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:05:13.348394 6056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:05:13.348553 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:43923
--local_ip_for_outbound_sockets=127.21.66.67
--tserver_master_addrs=127.21.66.126:39877
--webserver_port=34367
--webserver_interface=127.21.66.67
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:13.430966 6059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:13.431190 6059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:13.431217 6059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:13.432355 6059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:13.432392 6059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:13.432420 6059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:05:13.433948 6059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:43923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=34367
--tserver_master_addrs=127.21.66.126:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:13.434234 6059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:13.434437 6059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:13.439714 6065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:13.439699 6067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:13.439723 6064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:13.439972 6059 server_base.cc:1034] running on GCE node
I20260620 23:05:13.440269 6059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:13.440550 6059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:13.441756 6059 hybrid_clock.cc:648] HybridClock initialized: now 1781996713441719 us; error 49 us; skew 500 ppm
Jun 20 23:05:13 dist-test-slave-qgtk krb5kdc[4619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996713, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:13.444566 6059 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:05:13.445094 6059 webserver.cc:460] Webserver started at http://127.21.66.67:34367/ using document root <none> and password file <none>
I20260620 23:05:13.445341 6059 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:13.445411 6059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:13.446975 6059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:05:13.447781 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.448014 6059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:13.448107 6059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "f6605fa98d804ca096581fff7a544a85"
format_stamp: "Formatted at 2026-06-20 23:04:31 on dist-test-slave-qgtk"
server_key: "53b426998b480c43da076c344fe35854"
server_key_iv: "323a542636dfa4f6ff56807130376c15"
server_key_version: "encryptionkey@0"
I20260620 23:05:13.448485 6059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:13.488749 6059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:13.491240 6059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:13.491362 6059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:13.491539 6059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:13.491875 6059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:13.491914 6059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.491936 6059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:13.491982 6059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:13.497695 6059 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:43923
I20260620 23:05:13.497776 6187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:43923 every 8 connection(s)
I20260620 23:05:13.498167 6059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:13.499255 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.498876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:50817)
0620 23:05:13.499037 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.499109 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
W20260620 23:05:13.499660 6188 heartbeater.cc:644] Failed to heartbeat to 127.21.66.126:39877 (0 consecutive failures): Network error: Failed to ping master at 127.21.66.126:39877: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
I20260620 23:05:13.499971 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.499864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:60315)
0620 23:05:13.499917 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.499945 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 23:05:13.500247 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:13.500160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:37531)
0620 23:05:13.500204 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:13.500226 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:05:13.500360 6188 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:05:13.504554 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 6059
I20260620 23:05:14.289911 6192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.289641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:44453)
0620 23:05:14.289799 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:14.289846 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260620 23:05:14.349009 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.348745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:38175)
0620 23:05:14.348894 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:14.348936 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260620 23:05:14.500957 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.500699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:54391)
0620 23:05:14.500855 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:14.500900 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
I20260620 23:05:14.902767 6195 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:14.902909 6195 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:14.903251 6195 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
Jun 20 23:05:14 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996713, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.66@KRBTEST.COM
Jun 20 23:05:14 dist-test-slave-qgtk krb5kdc[4619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996713, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.67@KRBTEST.COM
I20260620 23:05:14.917135 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.903413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:38595 (local address 127.21.66.65:48757)
0620 23:05:14.903538 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:14.903552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:14.903623 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:14.904783 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:14.904786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:14.905389 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:14.905708 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:14.905716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.907534 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.907536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:14.909034 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:14.909036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.909977 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.909977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:14.910179 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:14.910851 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:14.910855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:14.912909 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:14.915346 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:14.915353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:14.915356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:14.915582 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:14.915878 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:14.915879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:14.915879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:14.916006 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:14.916470 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:14.916478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:14.916652 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 23:05:14.916778 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 23:05:14.916994 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":102,"thread_start_us":53,"threads_started":1}
I20260620 23:05:14.917204 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.903553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:48757 (local address 127.21.66.66:38595)
0620 23:05:14.903726 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:14.903739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:14.903766 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:05:14.904335 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:14.904339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:14.904401 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:14.904573 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:14.905990 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.907393 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.909133 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.909794 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.909904 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:14.913047 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:14.913051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:14.913052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:14.913065 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:14.915246 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:14.915690 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:14.915694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:14.915694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:14.915800 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:14.916076 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:14.916078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:14.916078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:14.916390 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:14.916474 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:14.916877 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 23:05:14.917023 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260620 23:05:14.917208 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.903413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:43923 (local address 127.21.66.65:43993)
0620 23:05:14.903538 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:14.903552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:14.903613 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:14.904790 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:14.904791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:14.905389 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:14.905708 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:14.905716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.907536 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.907537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:14.909028 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:14.909032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.909975 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.909977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:14.910179 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:14.910830 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:14.910841 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:14.913258 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:14.915702 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:14.915703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:14.915704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:14.915900 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:14.916198 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:14.916199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:14.916199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:14.916293 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:14.916827 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:14.916837 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:14.916889 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 23:05:14.916965 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 23:05:14.917065 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":89,"thread_start_us":59,"threads_started":1}
W20260620 23:05:14.917973 5809 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:05:14.918242 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:14.903553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:43993 (local address 127.21.66.67:43923)
0620 23:05:14.903726 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:14.903739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:14.903768 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:05:14.904341 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:14.904344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:14.904401 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:14.904568 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:14.905863 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.907394 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.909133 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:14.909784 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:14.909905 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:14.913369 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:14.913372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:14.913373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:14.913383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:14.915592 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:14.916026 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:14.916029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:14.916029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:14.916117 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:14.916406 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:14.916407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:14.916407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:14.916749 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:14.916835 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:14.917053 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:05:14.917194 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":56,"threads_started":1}
W20260620 23:05:14.918870 5810 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:05:14.918929 5810 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5204e8974fde46c7868d56c380da861c; no voters: 788cd9702e8144e9b2162aa59ca97cf5, f6605fa98d804ca096581fff7a544a85
I20260620 23:05:14.919042 6195 raft_consensus.cc:2726] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:05:15.290555 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:15.290386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:53819)
0620 23:05:15.290450 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:15.290501 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:05:15.349797 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:15.349529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:58081)
0620 23:05:15.349679 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:15.349744 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260620 23:05:15.501610 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:15.501422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:39425)
0620 23:05:15.501534 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:15.501567 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260620 23:05:16.291373 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:16.291124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:55207)
0620 23:05:16.291273 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:16.291322 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260620 23:05:16.350572 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:16.350331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:48849)
0620 23:05:16.350476 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:16.350517 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260620 23:05:16.502414 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:16.502202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:38747)
0620 23:05:16.502328 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:16.502369 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260620 23:05:17.073159 6205 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:17.073283 6205 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:17.073511 6205 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
W20260620 23:05:17.074214 5810 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
W20260620 23:05:17.074291 5809 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:05:17.074355 5809 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5204e8974fde46c7868d56c380da861c; no voters: 788cd9702e8144e9b2162aa59ca97cf5, f6605fa98d804ca096581fff7a544a85
I20260620 23:05:17.074436 6205 raft_consensus.cc:2726] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:05:17.292222 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:17.291910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:43073)
0620 23:05:17.292099 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:17.292151 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260620 23:05:17.351366 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:17.351129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:33177)
0620 23:05:17.351269 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:17.351308 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260620 23:05:17.503187 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:17.502921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:58013)
0620 23:05:17.503078 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:17.503128 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260620 23:05:18.293108 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:18.292858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:57655)
0620 23:05:18.293008 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:18.293057 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260620 23:05:18.352047 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:18.351848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:48947)
0620 23:05:18.351965 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:18.352001 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260620 23:05:18.504108 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:18.503858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:42799)
0620 23:05:18.504010 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:18.504060 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260620 23:05:19.294023 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:19.293755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:36831)
0620 23:05:19.293923 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:19.293974 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260620 23:05:19.352883 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:19.352626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:41673)
0620 23:05:19.352790 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:19.352834 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260620 23:05:19.504937 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:19.504676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:50609)
0620 23:05:19.504839 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:19.504889 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260620 23:05:19.607685 6215 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:19.607796 6215 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:19.608048 6215 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
W20260620 23:05:19.608573 5810 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
W20260620 23:05:19.608647 5809 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:05:19.608680 5809 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5204e8974fde46c7868d56c380da861c; no voters: 788cd9702e8144e9b2162aa59ca97cf5, f6605fa98d804ca096581fff7a544a85
I20260620 23:05:19.608783 6215 raft_consensus.cc:2726] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:05:20.294978 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:20.294660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:42233)
0620 23:05:20.294884 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:20.294930 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260620 23:05:20.353870 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:20.353593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:54915)
0620 23:05:20.353775 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:20.353822 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260620 23:05:20.505739 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:20.505497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:56507)
0620 23:05:20.505639 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:20.505688 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260620 23:05:21.295779 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:21.295531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:34927)
0620 23:05:21.295681 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:21.295728 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260620 23:05:21.336830 6220 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:21.336954 6220 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:21.337163 6220 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
W20260620 23:05:21.337677 5809 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
W20260620 23:05:21.337769 5810 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:05:21.337889 5810 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5204e8974fde46c7868d56c380da861c; no voters: 788cd9702e8144e9b2162aa59ca97cf5, f6605fa98d804ca096581fff7a544a85
I20260620 23:05:21.337963 6220 raft_consensus.cc:2726] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:05:21.354697 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:21.354436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:53985)
0620 23:05:21.354595 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:21.354646 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260620 23:05:21.506621 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:21.506396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:42701)
0620 23:05:21.506541 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:21.506576 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260620 23:05:22.296774 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:22.296504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:49359)
0620 23:05:22.296673 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:22.296722 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260620 23:05:22.355677 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:22.355429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:46365)
0620 23:05:22.355581 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:22.355629 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260620 23:05:22.507365 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:22.507127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:45231)
0620 23:05:22.507264 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:22.507316 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260620 23:05:23.297742 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:23.297500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:37551)
0620 23:05:23.297646 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:23.297695 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260620 23:05:23.356432 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:23.356157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:33445)
0620 23:05:23.356348 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:23.356385 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
I20260620 23:05:23.508170 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:23.507921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:45511)
0620 23:05:23.508076 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:23.508119 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260620 23:05:24.298766 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.298444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:53327)
0620 23:05:24.298672 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:24.298719 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":135,"threads_started":1}
I20260620 23:05:24.357240 6230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.356980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:59895)
0620 23:05:24.357142 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:24.357190 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260620 23:05:24.509194 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:24.508911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:43013)
0620 23:05:24.509085 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:24.509132 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260620 23:05:25.299749 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:25.299370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:54727)
0620 23:05:25.299573 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:25.299621 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260620 23:05:25.358106 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:25.357848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:33341)
0620 23:05:25.358008 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:25.358055 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260620 23:05:25.510012 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:25.509765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:50559)
0620 23:05:25.509915 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:25.509963 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260620 23:05:26.300635 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:26.300349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:43829)
0620 23:05:26.300502 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:26.300544 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260620 23:05:26.359215 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:26.358886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:58767)
0620 23:05:26.359116 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:26.359164 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
I20260620 23:05:26.510916 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:26.510598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:49513)
0620 23:05:26.510813 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:26.510860 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":120,"threads_started":1}
I20260620 23:05:26.574601 6238 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:26.574723 6238 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:26.574923 6238 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
W20260620 23:05:26.575313 5810 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
W20260620 23:05:26.575519 5809 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:05:26.575570 5809 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5204e8974fde46c7868d56c380da861c; no voters: 788cd9702e8144e9b2162aa59ca97cf5, f6605fa98d804ca096581fff7a544a85
I20260620 23:05:26.575654 6238 raft_consensus.cc:2726] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:05:27.301491 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.301217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:56003)
0620 23:05:27.301382 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:27.301431 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
I20260620 23:05:27.360211 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.359936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:54187)
0620 23:05:27.360114 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:27.360163 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":123,"threads_started":1}
I20260620 23:05:27.511714 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:27.511486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:46161)
0620 23:05:27.511627 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:27.511667 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260620 23:05:28.302412 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:28.302166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.65:55089)
0620 23:05:28.302315 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:28.302361 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260620 23:05:28.361091 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:28.360871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.66:55741)
0620 23:05:28.361002 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:28.361044 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260620 23:05:28.362514 6244 raft_consensus.cc:482] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:28.362641 6244 raft_consensus.cc:504] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6605fa98d804ca096581fff7a544a85" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 43923 } } peers { permanent_uuid: "788cd9702e8144e9b2162aa59ca97cf5" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 38595 } } peers { permanent_uuid: "5204e8974fde46c7868d56c380da861c" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42413 } }
I20260620 23:05:28.362877 6244 leader_election.cc:288] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923), 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595)
W20260620 23:05:28.363320 5810 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6605fa98d804ca096581fff7a544a85 (127.21.66.67:43923): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
W20260620 23:05:28.363440 5809 leader_election.cc:341] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 788cd9702e8144e9b2162aa59ca97cf5 (127.21.66.66:38595): Not found: Tablet not found: 6fc302f90d1a4e9b84420eb1c74eb2a5
I20260620 23:05:28.363529 5809 leader_election.cc:302] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5204e8974fde46c7868d56c380da861c; no voters: 788cd9702e8144e9b2162aa59ca97cf5, f6605fa98d804ca096581fff7a544a85
I20260620 23:05:28.363623 6244 raft_consensus.cc:2726] T 6fc302f90d1a4e9b84420eb1c74eb2a5 P 5204e8974fde46c7868d56c380da861c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:05:28.509573 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 5758
I20260620 23:05:28.512586 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:28.512334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39877 (local address 127.21.66.67:37085)
0620 23:05:28.512499 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:28.512536 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.126:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260620 23:05:28.514946 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 5926
I20260620 23:05:28.519279 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 6059
2026-06-20T23:05:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58373 ms)
[----------] 30 tests from SecurityITest (240924 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:28 dist-test-slave-qgtk krb5kdc[6248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:28 dist-test-slave-qgtk krb5kdc[6248](info): set up 2 sockets
Jun 20 23:05:28 dist-test-slave-qgtk krb5kdc[6248](info): commencing operation
krb5kdc: starting...
W20260620 23:05:29.586493 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:29 dist-test-slave-qgtk krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:05:29.714082 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:35153
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:35153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:29.796051 6261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:29.796298 6261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:29.796334 6261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:29.797489 6261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:29.797559 6261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:29.797585 6261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:29.797607 6261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:29.797636 6261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:29.799161 6261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:35153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:35153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:29.799448 6261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:29.799687 6261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:29.805042 6267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:29.805091 6261 server_base.cc:1034] running on GCE node
W20260620 23:05:29.805045 6266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:29.805052 6269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:29.805527 6261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:29.805861 6261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:29.807040 6261 hybrid_clock.cc:648] HybridClock initialized: now 1781996729807028 us; error 46 us; skew 500 ppm
Jun 20 23:05:29 dist-test-slave-qgtk krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:29.809688 6261 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:29.810096 6261 webserver.cc:460] Webserver started at http://127.21.66.126:35845/ using document root <none> and password file <none>
I20260620 23:05:29.810420 6261 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:29.810469 6261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:29.810566 6261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:29.811621 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "74b949f97dfa43aaaf722df7e5da2142"
format_stamp: "Formatted at 2026-06-20 23:05:29 on dist-test-slave-qgtk"
server_key: "31cc73035953e3a50a3cc862176e68a6"
server_key_iv: "dd2b48983a5ca53cb384a42f3e3ab2e0"
server_key_version: "encryptionkey@0"
I20260620 23:05:29.811976 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "74b949f97dfa43aaaf722df7e5da2142"
format_stamp: "Formatted at 2026-06-20 23:05:29 on dist-test-slave-qgtk"
server_key: "31cc73035953e3a50a3cc862176e68a6"
server_key_iv: "dd2b48983a5ca53cb384a42f3e3ab2e0"
server_key_version: "encryptionkey@0"
I20260620 23:05:29.813335 6261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:29.814164 6276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:29.814381 6261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:29.814505 6261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "74b949f97dfa43aaaf722df7e5da2142"
format_stamp: "Formatted at 2026-06-20 23:05:29 on dist-test-slave-qgtk"
server_key: "31cc73035953e3a50a3cc862176e68a6"
server_key_iv: "dd2b48983a5ca53cb384a42f3e3ab2e0"
server_key_version: "encryptionkey@0"
I20260620 23:05:29.814597 6261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:29.826898 6261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:29.829396 6261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:29.829583 6261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:29.833657 6261 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:35153
I20260620 23:05:29.833691 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:35153 every 8 connection(s)
I20260620 23:05:29.834053 6261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:29.834972 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:29.837311 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142: Bootstrap starting.
I20260620 23:05:29.837913 6329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:29.838267 6329 log.cc:824] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:29.839061 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142: No bootstrap required, opened a new log
I20260620 23:05:29.839354 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 6261
I20260620 23:05:29.839463 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:29.839665 21769 external_mini_cluster.cc:1428] Setting key 1be659297379c98f2016e2483d44428c
I20260620 23:05:29.841670 6329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b949f97dfa43aaaf722df7e5da2142" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35153 } }
I20260620 23:05:29.841847 6329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:29.841951 6329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b949f97dfa43aaaf722df7e5da2142, State: Initialized, Role: FOLLOWER
I20260620 23:05:29.842092 6329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b949f97dfa43aaaf722df7e5da2142" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35153 } }
I20260620 23:05:29.842335 6329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:29.842418 6329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:29.842474 6329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:29.843421 6329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b949f97dfa43aaaf722df7e5da2142" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35153 } }
I20260620 23:05:29.843552 6329 leader_election.cc:302] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b949f97dfa43aaaf722df7e5da2142; no voters:
I20260620 23:05:29.843732 6329 leader_election.cc:288] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:29.843909 6334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:29.844106 6334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [term 1 LEADER]: Becoming Leader. State: Replica: 74b949f97dfa43aaaf722df7e5da2142, State: Running, Role: LEADER
I20260620 23:05:29.844108 6329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:29.844256 6334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b949f97dfa43aaaf722df7e5da2142" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35153 } }
I20260620 23:05:29.844738 6335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74b949f97dfa43aaaf722df7e5da2142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b949f97dfa43aaaf722df7e5da2142" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35153 } } }
I20260620 23:05:29.844820 6335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:29.845000 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b949f97dfa43aaaf722df7e5da2142. Latest consensus state: current_term: 1 leader_uuid: "74b949f97dfa43aaaf722df7e5da2142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b949f97dfa43aaaf722df7e5da2142" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35153 } } }
I20260620 23:05:29.845080 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:29.845484 6341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:29.845988 6341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:29.847812 6341 catalog_manager.cc:1347] Generated new cluster ID: 0e108db3d5a34397bbf0e999fa343eaa
I20260620 23:05:29.847878 6341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:05:29 dist-test-slave-qgtk krb5kdc[6248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:29.853420 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:29.840209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.21.66.126:35153)
0620 23:05:29.840412 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:29.840423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:29.840456 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:05:29.841616 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:29.841625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:29.841692 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:29.841853 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:29.842708 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:29.844263 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:29.845620 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:29.846401 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:29.846554 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:29.849438 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:29.849442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:29.849443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:29.849456 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:29.851572 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:29.852024 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:29.852027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:29.852028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:29.852132 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:29.852380 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:29.852381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:29.852381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:29.852696 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:29.852929 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:29.853056 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 23:05:29.853227 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260620 23:05:29.874650 6341 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:29.875272 6341 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:29.878933 6341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142: Generated new TSK 0
I20260620 23:05:29.879225 6341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:05:29.917053 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:35153
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:30.026929 6355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.027180 6355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.027220 6355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.028463 6355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.028550 6355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.028595 6355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:05:30.030089 6355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:35153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:30.030393 6355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.030632 6355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.036099 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.036106 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.036113 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.036142 6355 server_base.cc:1034] running on GCE node
I20260620 23:05:30.036500 6355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.036801 6355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.037967 6355 hybrid_clock.cc:648] HybridClock initialized: now 1781996730037916 us; error 62 us; skew 500 ppm
Jun 20 23:05:30 dist-test-slave-qgtk krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:30.041024 6355 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:05:30.041594 6355 webserver.cc:460] Webserver started at http://127.21.66.65:42559/ using document root <none> and password file <none>
I20260620 23:05:30.041806 6355 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.041848 6355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.041940 6355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.043128 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "f0a0b75695a24aefb922dbb86468e1e8"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "4b93300177e9d38c645c5d8e4282330d"
server_key_iv: "0a4dd0eb0e996d3745fe87c29ad7fac2"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.043439 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "f0a0b75695a24aefb922dbb86468e1e8"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "4b93300177e9d38c645c5d8e4282330d"
server_key_iv: "0a4dd0eb0e996d3745fe87c29ad7fac2"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.044960 6355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:05:30.045815 6370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.046312 6355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:30.046420 6355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "f0a0b75695a24aefb922dbb86468e1e8"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "4b93300177e9d38c645c5d8e4282330d"
server_key_iv: "0a4dd0eb0e996d3745fe87c29ad7fac2"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.046486 6355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:30.066584 6355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:30.069245 6355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:30.069423 6355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:30.069638 6355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:30.070053 6355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:30.070094 6355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.070169 6355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:30.070197 6355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.076349 6355 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:44531
I20260620 23:05:30.076447 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:44531 every 8 connection(s)
I20260620 23:05:30.076777 6355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:30.083267 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 6355
I20260620 23:05:30.083456 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:30.083679 21769 external_mini_cluster.cc:1428] Setting key 61b91a2b5dc3f9a64e7677a468a81927
Jun 20 23:05:30 dist-test-slave-qgtk krb5kdc[6248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:30.088019 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.077689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:37329 (local address 127.21.66.126:35153)
0620 23:05:30.077759 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.077761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.078062 (+ 301us) server_negotiation.cc:407] Connection header received
0620 23:05:30.078707 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.078708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.078719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.078780 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.079926 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.080542 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.081616 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.081826 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.081889 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.084566 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.084567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.084567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:30.084575 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.086383 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.086936 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.086937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.086937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.086990 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.087338 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.087338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.087339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.087453 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.087506 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.087874 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.087950 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:05:30.088380 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.077660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:35153 (local address 127.21.66.65:37329)
0620 23:05:30.077926 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.077964 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.078559 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.078930 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.078934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.079540 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:30.079819 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.079824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.080635 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.080636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.081516 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.081518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.081912 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.081912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.082022 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.082650 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:30.082660 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:30.084492 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:30.086468 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.086476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.086479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.086847 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.087068 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.087070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.087070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.087238 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.087523 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:30.087530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:30.087723 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.087850 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.088052 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":165,"threads_started":1}
I20260620 23:05:30.089053 6484 heartbeater.cc:347] Connected to a master server at 127.21.66.126:35153
I20260620 23:05:30.089184 6484 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:30.089370 6484 heartbeater.cc:511] Master 127.21.66.126:35153 requested a full tablet report, sending...
I20260620 23:05:30.090030 6293 ts_manager.cc:194] Registered new tserver with Master: f0a0b75695a24aefb922dbb86468e1e8 (127.21.66.65:44531)
I20260620 23:05:30.091151 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:37329
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:05:30.112939 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:35153
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:30.195580 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.195807 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.195833 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.197062 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.197108 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.197139 6489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:05:30.198689 6489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:35153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:30.198969 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.199210 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.204773 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.204887 6489 server_base.cc:1034] running on GCE node
W20260620 23:05:30.204782 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.204773 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.205225 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.205549 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.206707 6489 hybrid_clock.cc:648] HybridClock initialized: now 1781996730206671 us; error 46 us; skew 500 ppm
Jun 20 23:05:30 dist-test-slave-qgtk krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:30.209311 6489 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:05:30.209858 6489 webserver.cc:460] Webserver started at http://127.21.66.66:46781/ using document root <none> and password file <none>
I20260620 23:05:30.210155 6489 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.210237 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.210448 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.211506 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "e8a44233107f444ea02ecb44f715783a"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "8b3990f44d6f302d093f271ce6b69377"
server_key_iv: "8a829564cfdb84c3e729026e903a7b07"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.211875 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "e8a44233107f444ea02ecb44f715783a"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "8b3990f44d6f302d093f271ce6b69377"
server_key_iv: "8a829564cfdb84c3e729026e903a7b07"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.213249 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:30.214033 6504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.214293 6489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:30.214413 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "e8a44233107f444ea02ecb44f715783a"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "8b3990f44d6f302d093f271ce6b69377"
server_key_iv: "8a829564cfdb84c3e729026e903a7b07"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.214514 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:30.227909 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:30.230461 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:30.230636 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:30.230896 6489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:30.231253 6489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:30.231315 6489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.231364 6489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:30.231398 6489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.238032 6489 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:33415
I20260620 23:05:30.238184 6617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:33415 every 8 connection(s)
I20260620 23:05:30.238511 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
Jun 20 23:05:30 dist-test-slave-qgtk krb5kdc[6248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:30.248225 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 6489
I20260620 23:05:30.248365 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:30.248576 21769 external_mini_cluster.cc:1428] Setting key a113bade67451a0723150d36cc9cb95d
I20260620 23:05:30.249166 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.239362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:53555 (local address 127.21.66.126:35153)
0620 23:05:30.239470 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.239473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.239664 (+ 191us) server_negotiation.cc:407] Connection header received
0620 23:05:30.240352 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.240353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.240367 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.240423 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.241500 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.242224 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.243371 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.243602 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.243706 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.246292 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.246294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.246294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:30.246302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.247556 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.247980 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.247981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.247982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.248047 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.248344 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.248345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.248346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.248541 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.248616 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.248982 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.249066 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:05:30.249421 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.239274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:35153 (local address 127.21.66.66:53555)
0620 23:05:30.239514 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.239571 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.240212 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.240498 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.240509 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.241160 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:30.241387 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.241392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.242326 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.242327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.243212 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.243214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.243736 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.243737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.243881 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.244497 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:30.244507 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:30.246187 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:30.247641 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.247649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.247652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.247881 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.248128 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.248129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.248130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.248249 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.248637 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:30.248651 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:30.248774 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.248959 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.249095 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
I20260620 23:05:30.250144 6618 heartbeater.cc:347] Connected to a master server at 127.21.66.126:35153
I20260620 23:05:30.250250 6618 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:30.250447 6618 heartbeater.cc:511] Master 127.21.66.126:35153 requested a full tablet report, sending...
I20260620 23:05:30.250818 6293 ts_manager.cc:194] Registered new tserver with Master: e8a44233107f444ea02ecb44f715783a (127.21.66.66:33415)
I20260620 23:05:30.251247 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:53555
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:05:30.273463 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:35153
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:44419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:30.360637 6623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.360888 6623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.360917 6623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.362298 6623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.362352 6623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.362383 6623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:05:30.363906 6623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:44419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:35153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:30.364123 6623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.364328 6623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.369717 6628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.369670 6631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.369921 6623 server_base.cc:1034] running on GCE node
W20260620 23:05:30.369679 6629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.370244 6623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.370532 6623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.371695 6623 hybrid_clock.cc:648] HybridClock initialized: now 1781996730371686 us; error 42 us; skew 500 ppm
Jun 20 23:05:30 dist-test-slave-qgtk krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:30.374545 6623 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:05:30.375013 6623 webserver.cc:460] Webserver started at http://127.21.66.67:32977/ using document root <none> and password file <none>
I20260620 23:05:30.375263 6623 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.375342 6623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.375486 6623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.376586 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "440d7089998d454995d7ddc0e5c165db"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "fb4ae65deb12991f246d3de2d8393343"
server_key_iv: "73a464366ec6c4acffbaa09419ef402d"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.376946 6623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "440d7089998d454995d7ddc0e5c165db"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "fb4ae65deb12991f246d3de2d8393343"
server_key_iv: "73a464366ec6c4acffbaa09419ef402d"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.378319 6623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:30.379348 6638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.379537 6623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:30.379633 6623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "440d7089998d454995d7ddc0e5c165db"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "fb4ae65deb12991f246d3de2d8393343"
server_key_iv: "73a464366ec6c4acffbaa09419ef402d"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.379722 6623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:30.390077 6623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:30.392628 6623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:30.392792 6623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:30.393028 6623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:30.393342 6623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:30.393399 6623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.393441 6623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:30.393483 6623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.399399 6623 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:45075
I20260620 23:05:30.399482 6751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:45075 every 8 connection(s)
I20260620 23:05:30.399803 6623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:05:30 dist-test-slave-qgtk krb5kdc[6248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996730, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:30.409229 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 6623
I20260620 23:05:30.409425 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:30.409659 21769 external_mini_cluster.cc:1428] Setting key d160cc77c138b3350e4717c8f2131969
I20260620 23:05:30.410418 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.400588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:44293 (local address 127.21.66.126:35153)
0620 23:05:30.400631 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.400633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.400859 (+ 226us) server_negotiation.cc:407] Connection header received
0620 23:05:30.401422 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.401423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.401437 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.401494 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.402595 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.403110 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.404125 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.404324 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.404413 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.406825 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.406826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.406827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:30.406837 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.408396 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.408869 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.408871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.408871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.408932 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.409365 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.409367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.409368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.409582 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.409658 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.410092 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.410284 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:05:30.410557 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.400543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:35153 (local address 127.21.66.67:44293)
0620 23:05:30.400728 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.400764 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.401303 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.401566 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.401574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.402264 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:30.402492 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.402498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.403209 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.403210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.404022 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.404024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.404435 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.404436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.404571 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.405089 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:30.405097 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:30.406708 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:30.408491 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.408499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.408501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.408772 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.409058 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:30.409059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:30.409060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:30.409221 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:30.409660 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:30.409667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:30.409935 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.410036 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.410310 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
I20260620 23:05:30.411221 6752 heartbeater.cc:347] Connected to a master server at 127.21.66.126:35153
I20260620 23:05:30.411374 6752 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:30.411581 6752 heartbeater.cc:511] Master 127.21.66.126:35153 requested a full tablet report, sending...
I20260620 23:05:30.412083 6293 ts_manager.cc:194] Registered new tserver with Master: 440d7089998d454995d7ddc0e5c165db (127.21.66.67:45075)
I20260620 23:05:30.412449 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:44293
I20260620 23:05:30.422353 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:30.430032 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.423773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.21.66.126:35153)
0620 23:05:30.423856 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.423859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.423951 (+ 92us) server_negotiation.cc:407] Connection header received
0620 23:05:30.423995 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.423996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.424007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.424075 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.424649 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.425185 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.425942 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.426192 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.426252 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.427518 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.427519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.427519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:30.427526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.429026 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.429360 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.429361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.429361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.429436 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:30.429598 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:30.429598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:30.429599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:30.429742 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.429815 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.429907 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.429965 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:05:30.431756 6293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:30.433373 6293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:30.440770 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.436156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.21.66.66:33415)
0620 23:05:30.436364 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.436369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.436380 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:05:30.436487 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.436490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.436506 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.436561 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:30.436900 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.438138 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.439629 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.440263 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.440346 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.440416 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.440628 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.440674 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
I20260620 23:05:30.440783 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.435940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:33415 (local address 127.0.0.1:34284)
0620 23:05:30.436338 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.436348 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.436386 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.436692 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.436695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.436698 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:30.436773 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.436775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.438279 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.438279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.439536 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.439539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.440371 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.440372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.440487 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.440505 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.440609 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.440674 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":120,"thread_start_us":81,"threads_started":1}
I20260620 23:05:30.441304 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.435940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:45075 (local address 127.0.0.1:43640)
0620 23:05:30.436111 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.436155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.436219 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.436879 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.436880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.436883 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:30.436952 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.436954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.438277 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.438278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.439952 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.439956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.441018 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.441019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.441085 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.441090 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.441145 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.441197 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":159,"thread_start_us":63,"threads_started":1}
I20260620 23:05:30.441555 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.436390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.21.66.67:45075)
0620 23:05:30.436563 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.436568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.436584 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:05:30.436636 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.436638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.436657 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.436755 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:30.437049 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.438170 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.440065 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.440844 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.440918 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.440982 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.441200 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.441253 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260620 23:05:30.442034 6552 tablet_service.cc:1467] Processing CreateTablet for tablet 412a74e3b22244d3bd60d2305240c1bb (DEFAULT_TABLE table=test-table [id=ac1d392a25ba4b1ab993649f16d5a63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:30.442480 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.436147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:44531 (local address 127.0.0.1:35876)
0620 23:05:30.436587 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.436601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.436793 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.437164 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.437165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.437167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:30.437209 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.437210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.438281 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.438283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.439955 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.439956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.441174 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.441175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.441246 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.441251 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.441317 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.441364 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":130,"thread_start_us":68,"threads_started":1}
I20260620 23:05:30.442552 6552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412a74e3b22244d3bd60d2305240c1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:30.442927 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.436309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.21.66.65:44531)
0620 23:05:30.436474 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.436478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.436906 (+ 428us) server_negotiation.cc:407] Connection header received
0620 23:05:30.436960 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.436963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.436980 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.437066 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:30.437306 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.438106 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.440107 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.440966 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.441060 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.441127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.442712 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.442862 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260620 23:05:30.442966 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 412a74e3b22244d3bd60d2305240c1bb (DEFAULT_TABLE table=test-table [id=ac1d392a25ba4b1ab993649f16d5a63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:30.443502 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412a74e3b22244d3bd60d2305240c1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:30.447057 6769 tablet_bootstrap.cc:492] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: Bootstrap starting.
I20260620 23:05:30.447149 6770 tablet_bootstrap.cc:492] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Bootstrap starting.
I20260620 23:05:30.447767 6418 tablet_service.cc:1467] Processing CreateTablet for tablet 412a74e3b22244d3bd60d2305240c1bb (DEFAULT_TABLE table=test-table [id=ac1d392a25ba4b1ab993649f16d5a63b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:30.447875 6770 tablet_bootstrap.cc:654] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:30.448109 6418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412a74e3b22244d3bd60d2305240c1bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:30.448566 6770 log.cc:824] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:30.449378 6770 tablet_bootstrap.cc:492] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: No bootstrap required, opened a new log
I20260620 23:05:30.449470 6770 ts_tablet_manager.cc:1397] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:30.451089 6772 tablet_bootstrap.cc:492] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8: Bootstrap starting.
I20260620 23:05:30.451205 6770 raft_consensus.cc:348] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.451426 6770 raft_consensus.cc:374] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:30.451479 6770 raft_consensus.cc:729] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 440d7089998d454995d7ddc0e5c165db, State: Initialized, Role: FOLLOWER
I20260620 23:05:30.451578 6769 tablet_bootstrap.cc:654] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:30.451650 6770 consensus_queue.cc:260] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.451804 6772 tablet_bootstrap.cc:654] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:30.451987 6770 ts_tablet_manager.cc:1428] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 23:05:30.452129 6752 heartbeater.cc:503] Master 127.21.66.126:35153 was elected leader, sending a full tablet report...
I20260620 23:05:30.452245 6769 log.cc:824] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:30.452531 6772 log.cc:824] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:30.453246 6769 tablet_bootstrap.cc:492] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: No bootstrap required, opened a new log
I20260620 23:05:30.453320 6769 ts_tablet_manager.cc:1397] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 23:05:30.453356 6772 tablet_bootstrap.cc:492] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8: No bootstrap required, opened a new log
I20260620 23:05:30.453428 6772 ts_tablet_manager.cc:1397] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:30.454890 6769 raft_consensus.cc:348] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.455057 6769 raft_consensus.cc:374] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:30.455106 6769 raft_consensus.cc:729] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a44233107f444ea02ecb44f715783a, State: Initialized, Role: FOLLOWER
I20260620 23:05:30.455235 6769 consensus_queue.cc:260] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.455152 6772 raft_consensus.cc:348] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.455336 6772 raft_consensus.cc:374] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:30.455411 6772 raft_consensus.cc:729] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a0b75695a24aefb922dbb86468e1e8, State: Initialized, Role: FOLLOWER
I20260620 23:05:30.455561 6769 ts_tablet_manager.cc:1428] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:05:30.455538 6772 consensus_queue.cc:260] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.455601 6618 heartbeater.cc:503] Master 127.21.66.126:35153 was elected leader, sending a full tablet report...
I20260620 23:05:30.455827 6772 ts_tablet_manager.cc:1428] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:05:30.455832 6484 heartbeater.cc:503] Master 127.21.66.126:35153 was elected leader, sending a full tablet report...
I20260620 23:05:30.487890 6777 raft_consensus.cc:482] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:30.488015 6777 raft_consensus.cc:504] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.488488 6777 leader_election.cc:288] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8a44233107f444ea02ecb44f715783a (127.21.66.66:33415), 440d7089998d454995d7ddc0e5c165db (127.21.66.67:45075)
W20260620 23:05:30.489157 6619 tablet.cc:2369] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:30.492031 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.488695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:45075 (local address 127.21.66.65:46495)
0620 23:05:30.488758 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.488781 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.488827 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.489071 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.489073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.489074 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:30.489146 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.489148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.489891 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.489892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.490869 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.490872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.491728 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.491729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.491808 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.491812 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.491892 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.491939 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:05:30.492280 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.488765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:46495 (local address 127.21.66.67:45075)
0620 23:05:30.488809 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.488818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.488920 (+ 102us) server_negotiation.cc:407] Connection header received
0620 23:05:30.488955 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.488956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.488966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.489036 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:30.489220 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.489806 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.490964 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.491631 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.491725 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.491736 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.492142 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.492192 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:05:30.492343 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.488765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:57667 (local address 127.21.66.66:33415)
0620 23:05:30.488809 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.488811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.489097 (+ 286us) server_negotiation.cc:407] Connection header received
0620 23:05:30.489126 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.489126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.489138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.489188 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:30.489530 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.490396 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.491408 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.491875 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.491934 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.491942 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.492213 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.492260 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:05:30.492585 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.488695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:33415 (local address 127.21.66.65:57667)
0620 23:05:30.488921 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.488933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.488994 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.489322 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.489323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.489325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:30.489386 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.489388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.490481 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.490482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.491318 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.491320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.491974 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.491975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.492043 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.492047 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.492425 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.492478 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":24,"thread_start_us":132,"threads_started":1}
I20260620 23:05:30.492609 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a74e3b22244d3bd60d2305240c1bb" candidate_uuid: "f0a0b75695a24aefb922dbb86468e1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440d7089998d454995d7ddc0e5c165db" is_pre_election: true
I20260620 23:05:30.492776 6706 raft_consensus.cc:2454] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a0b75695a24aefb922dbb86468e1e8 in term 0.
I20260620 23:05:30.492964 6572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a74e3b22244d3bd60d2305240c1bb" candidate_uuid: "f0a0b75695a24aefb922dbb86468e1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a44233107f444ea02ecb44f715783a" is_pre_election: true
I20260620 23:05:30.493090 6572 raft_consensus.cc:2454] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a0b75695a24aefb922dbb86468e1e8 in term 0.
I20260620 23:05:30.493059 6373 leader_election.cc:302] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 440d7089998d454995d7ddc0e5c165db, f0a0b75695a24aefb922dbb86468e1e8; no voters:
I20260620 23:05:30.493249 6777 raft_consensus.cc:2781] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:30.493311 6777 raft_consensus.cc:482] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:30.493348 6777 raft_consensus.cc:3037] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:30.494102 6777 raft_consensus.cc:504] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.494289 6777 leader_election.cc:288] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [CANDIDATE]: Term 1 election: Requested vote from peers e8a44233107f444ea02ecb44f715783a (127.21.66.66:33415), 440d7089998d454995d7ddc0e5c165db (127.21.66.67:45075)
I20260620 23:05:30.494534 6572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a74e3b22244d3bd60d2305240c1bb" candidate_uuid: "f0a0b75695a24aefb922dbb86468e1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a44233107f444ea02ecb44f715783a"
I20260620 23:05:30.494552 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a74e3b22244d3bd60d2305240c1bb" candidate_uuid: "f0a0b75695a24aefb922dbb86468e1e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "440d7089998d454995d7ddc0e5c165db"
I20260620 23:05:30.494625 6572 raft_consensus.cc:3037] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:30.494643 6706 raft_consensus.cc:3037] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:30.495563 6572 raft_consensus.cc:2454] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a0b75695a24aefb922dbb86468e1e8 in term 1.
I20260620 23:05:30.495693 6706 raft_consensus.cc:2454] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a0b75695a24aefb922dbb86468e1e8 in term 1.
I20260620 23:05:30.495755 6374 leader_election.cc:302] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8a44233107f444ea02ecb44f715783a, f0a0b75695a24aefb922dbb86468e1e8; no voters:
I20260620 23:05:30.495852 6777 raft_consensus.cc:2781] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:30.496031 6777 raft_consensus.cc:686] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 1 LEADER]: Becoming Leader. State: Replica: f0a0b75695a24aefb922dbb86468e1e8, State: Running, Role: LEADER
I20260620 23:05:30.496135 6777 consensus_queue.cc:237] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } }
I20260620 23:05:30.496989 6293 catalog_manager.cc:5495] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a0b75695a24aefb922dbb86468e1e8 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "f0a0b75695a24aefb922dbb86468e1e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a0b75695a24aefb922dbb86468e1e8" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 44531 } health_report { overall_health: HEALTHY } } }
I20260620 23:05:30.515229 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.512364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.21.66.65:44531)
0620 23:05:30.512439 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.512441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.512449 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:05:30.512486 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.512486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.512509 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.512575 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:30.512755 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.513258 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.514198 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.514455 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.514521 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.514885 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:30.515001 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:30.515096 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.515115 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.515136 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:05:30.517377 6706 raft_consensus.cc:1261] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 1 FOLLOWER]: Refusing update from remote peer f0a0b75695a24aefb922dbb86468e1e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:30.517370 6572 raft_consensus.cc:1261] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 1 FOLLOWER]: Refusing update from remote peer f0a0b75695a24aefb922dbb86468e1e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:30.517776 6777 consensus_queue.cc:1035] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "440d7089998d454995d7ddc0e5c165db" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 45075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:30.517978 6779 consensus_queue.cc:1035] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a44233107f444ea02ecb44f715783a" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:30.520633 6782 mvcc.cc:205] Tried to move back new op lower bound from 7299058608195547136 to 7299058608112709632. Current Snapshot: MvccSnapshot[applied={T|T < 7299058608195547136}]
I20260620 23:05:30.521036 6783 mvcc.cc:205] Tried to move back new op lower bound from 7299058608195547136 to 7299058608112709632. Current Snapshot: MvccSnapshot[applied={T|T < 7299058608195547136}]
I20260620 23:05:30.521749 6784 mvcc.cc:205] Tried to move back new op lower bound from 7299058608195547136 to 7299058608112709632. Current Snapshot: MvccSnapshot[applied={T|T < 7299058608195547136}]
I20260620 23:05:30.523838 6292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:30.523943 6292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:30.524755 6292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74b949f97dfa43aaaf722df7e5da2142: Sending DeleteTablet for 3 replicas of tablet 412a74e3b22244d3bd60d2305240c1bb
I20260620 23:05:30.525108 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 412a74e3b22244d3bd60d2305240c1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:30 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:43640
I20260620 23:05:30.525101 6418 tablet_service.cc:1514] Processing DeleteTablet for tablet 412a74e3b22244d3bd60d2305240c1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:30 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:35876
I20260620 23:05:30.525098 6552 tablet_service.cc:1514] Processing DeleteTablet for tablet 412a74e3b22244d3bd60d2305240c1bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:30 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:34284
I20260620 23:05:30.525530 6795 tablet_replica.cc:330] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8: stopping tablet replica
I20260620 23:05:30.525631 6795 raft_consensus.cc:2229] T 412a74e3b22244d3bd60d2305240c1bb P f0a0b75695a24aefb922dbb86468e1e8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:30.525624 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 6355
I20260620 23:05:30.530282 6793 tablet_replica.cc:330] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: stopping tablet replica
I20260620 23:05:30.530443 6793 raft_consensus.cc:2229] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:30.530550 6794 tablet_replica.cc:330] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: stopping tablet replica
I20260620 23:05:30.530591 6793 raft_consensus.cc:2258] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:30.530690 6794 raft_consensus.cc:2229] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:30.530817 6794 raft_consensus.cc:2258] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 23:05:30.531095 6280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:44531 (error 108)
I20260620 23:05:30.531262 6793 ts_tablet_manager.cc:1905] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:30.531352 6794 ts_tablet_manager.cc:1905] T 412a74e3b22244d3bd60d2305240c1bb P e8a44233107f444ea02ecb44f715783a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:30.531451 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 6489
I20260620 23:05:30.532166 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.531507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:44531 (local address 127.0.0.1:35888)
0620 23:05:30.532084 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.532128 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:44531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554}
W20260620 23:05:30.532375 6280 catalog_manager.cc:4584] TS f0a0b75695a24aefb922dbb86468e1e8 (127.21.66.65:44531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 412a74e3b22244d3bd60d2305240c1bb: Network error: Client connection negotiation failed: client connection to 127.21.66.65:44531: connect: Connection refused (error 111)
I20260620 23:05:30.533332 6793 ts_tablet_manager.cc:1918] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:30.533420 6793 log.cc:1196] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/412a74e3b22244d3bd60d2305240c1bb
I20260620 23:05:30.533671 6793 ts_tablet_manager.cc:1939] T 412a74e3b22244d3bd60d2305240c1bb P 440d7089998d454995d7ddc0e5c165db: Deleting consensus metadata
I20260620 23:05:30.534165 6279 catalog_manager.cc:4841] TS 440d7089998d454995d7ddc0e5c165db (127.21.66.67:45075): tablet 412a74e3b22244d3bd60d2305240c1bb (table test-table [id=ac1d392a25ba4b1ab993649f16d5a63b]) successfully deleted
I20260620 23:05:30.540377 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 6623
I20260620 23:05:30.540462 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.540321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:33415 (local address 127.0.0.1:34294)
0620 23:05:30.540371 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.540406 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.66:33415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 23:05:30.545537 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 6261
2026-06-20T23:05:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2026 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:30Z Disabled control of system clock
I20260620 23:05:30.573479 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40407
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:05:30.656286 6800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.656517 6800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.656539 6800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.657823 6800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:30.657866 6800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.657877 6800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.657891 6800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:30.657903 6800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:30.659439 6800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:40407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:30.659691 6800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.659994 6800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.665004 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.665004 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.665023 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.665282 6800 server_base.cc:1034] running on GCE node
I20260620 23:05:30.665520 6800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.665853 6800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.667028 6800 hybrid_clock.cc:648] HybridClock initialized: now 1781996730667019 us; error 41 us; skew 500 ppm
I20260620 23:05:30.668358 6800 webserver.cc:460] Webserver started at http://127.21.66.126:43437/ using document root <none> and password file <none>
I20260620 23:05:30.668710 6800 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.668802 6800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.668980 6800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.670200 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "e159b6bb05364fd084ae88b5689506a0"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "983e0b37980f2e3c8382c1a3c25fca39"
server_key_iv: "5f91f82e92b86977af057e80021ee2c5"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.670596 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "e159b6bb05364fd084ae88b5689506a0"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "983e0b37980f2e3c8382c1a3c25fca39"
server_key_iv: "5f91f82e92b86977af057e80021ee2c5"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.672117 6800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:05:30.673036 6814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.673269 6800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:30.673388 6800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "e159b6bb05364fd084ae88b5689506a0"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "983e0b37980f2e3c8382c1a3c25fca39"
server_key_iv: "5f91f82e92b86977af057e80021ee2c5"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.673470 6800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:30.689929 6800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:30.690392 6800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:30.690577 6800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:30.694854 6800 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:40407
I20260620 23:05:30.694886 6866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:40407 every 8 connection(s)
I20260620 23:05:30.695271 6800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:30.696316 6867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:30.698258 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 6800
I20260620 23:05:30.698393 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:30.698618 21769 external_mini_cluster.cc:1428] Setting key b214211db2250416a9a8eb89e875e013
I20260620 23:05:30.698840 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0: Bootstrap starting.
I20260620 23:05:30.699514 6867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:30.699817 6867 log.cc:824] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:30.700506 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0: No bootstrap required, opened a new log
I20260620 23:05:30.702320 6867 raft_consensus.cc:348] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e159b6bb05364fd084ae88b5689506a0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40407 } }
I20260620 23:05:30.702474 6867 raft_consensus.cc:374] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:30.702533 6867 raft_consensus.cc:729] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e159b6bb05364fd084ae88b5689506a0, State: Initialized, Role: FOLLOWER
I20260620 23:05:30.702692 6867 consensus_queue.cc:260] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e159b6bb05364fd084ae88b5689506a0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40407 } }
I20260620 23:05:30.702755 6867 raft_consensus.cc:388] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:30.702804 6867 raft_consensus.cc:482] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:30.702867 6867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:30.703393 6867 raft_consensus.cc:504] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e159b6bb05364fd084ae88b5689506a0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40407 } }
I20260620 23:05:30.703508 6867 leader_election.cc:302] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e159b6bb05364fd084ae88b5689506a0; no voters:
I20260620 23:05:30.703696 6867 leader_election.cc:288] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:30.703763 6872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:30.703975 6872 raft_consensus.cc:686] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [term 1 LEADER]: Becoming Leader. State: Replica: e159b6bb05364fd084ae88b5689506a0, State: Running, Role: LEADER
I20260620 23:05:30.703984 6867 sys_catalog.cc:564] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:30.704766 6872 consensus_queue.cc:237] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e159b6bb05364fd084ae88b5689506a0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40407 } }
I20260620 23:05:30.704939 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.699293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.21.66.126:40407)
0620 23:05:30.699568 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.699578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.699640 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:05:30.700160 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.700167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.700239 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.700404 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.700667 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.701660 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.702554 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.703159 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.703328 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.704100 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.704105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.704105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:30.704337 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.704380 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:30.704395 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.704460 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.704639 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.704727 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":117,"threads_started":1}
I20260620 23:05:30.705454 6873 sys_catalog.cc:455] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e159b6bb05364fd084ae88b5689506a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e159b6bb05364fd084ae88b5689506a0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40407 } } }
I20260620 23:05:30.705461 6874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e159b6bb05364fd084ae88b5689506a0. Latest consensus state: current_term: 1 leader_uuid: "e159b6bb05364fd084ae88b5689506a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e159b6bb05364fd084ae88b5689506a0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40407 } } }
I20260620 23:05:30.705895 6874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:30.705916 6873 sys_catalog.cc:458] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0 [sys.catalog]: This master's current role is: LEADER
W20260620 23:05:30.706544 6886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:30.706590 6886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:30.706663 6888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:30.706897 6888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:30.708418 6888 catalog_manager.cc:1347] Generated new cluster ID: e41f97f7a5d6459783e1a8615d30a1a0
I20260620 23:05:30.708477 6888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:30.724620 6888 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:30.725175 6888 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:30.728493 6888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0: Generated new TSK 0
I20260620 23:05:30.728675 6888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:30.730939 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40407
--builtin_ntp_servers=127.21.66.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:05:30.816879 6891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.817101 6891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.817127 6891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.818552 6891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.818600 6891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.818632 6891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:05:30.820210 6891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:30.820499 6891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.820839 6891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.827185 6897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.827172 6899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.827172 6896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.827438 6891 server_base.cc:1034] running on GCE node
I20260620 23:05:30.827667 6891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.827960 6891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.829141 6891 hybrid_clock.cc:648] HybridClock initialized: now 1781996730829103 us; error 53 us; skew 500 ppm
I20260620 23:05:30.830646 6891 webserver.cc:460] Webserver started at http://127.21.66.65:41075/ using document root <none> and password file <none>
I20260620 23:05:30.830919 6891 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.830988 6891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.831148 6891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.832286 6891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "255bcb4e2862483f9a6b07d7b9cbe031"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "593a29e0cee938dc0846987fc6581608"
server_key_iv: "2c42c954ae08431c4fc56c4df7ae264a"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.832664 6891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "255bcb4e2862483f9a6b07d7b9cbe031"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "593a29e0cee938dc0846987fc6581608"
server_key_iv: "2c42c954ae08431c4fc56c4df7ae264a"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.833983 6891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:30.834805 6905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.834980 6891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:30.835078 6891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "255bcb4e2862483f9a6b07d7b9cbe031"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "593a29e0cee938dc0846987fc6581608"
server_key_iv: "2c42c954ae08431c4fc56c4df7ae264a"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.835160 6891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:30.873062 6891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:30.873471 6891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:30.873629 6891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:30.873910 6891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:30.874396 6891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:30.874455 6891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.874498 6891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:30.874521 6891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.882695 6891 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:37305
I20260620 23:05:30.882773 7018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:37305 every 8 connection(s)
I20260620 23:05:30.883126 6891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:30.887158 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 6891
I20260620 23:05:30.887249 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:30.887458 21769 external_mini_cluster.cc:1428] Setting key 731003cae4c312f6226cb255ec723c22
I20260620 23:05:30.888506 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40407
--builtin_ntp_servers=127.21.66.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:05:30.888793 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.883851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:45939 (local address 127.21.66.126:40407)
0620 23:05:30.883952 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:30.883954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:30.884164 (+ 210us) server_negotiation.cc:407] Connection header received
0620 23:05:30.884734 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:30.884735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:30.884748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:30.884802 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:30.885485 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.885998 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.887066 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.887354 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.887445 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.888467 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:30.888469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:30.888469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:30.888473 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:30.888488 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:30.888492 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:30.888536 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:30.888654 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:05:30.888681 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:05:30.889279 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:30.883864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40407 (local address 127.21.66.65:45939)
0620 23:05:30.884045 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:30.884080 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:30.884616 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:30.884881 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:30.884884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:30.884893 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:30.885392 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.885396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.886098 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.886099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.886975 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:30.886978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:30.887457 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:30.887458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:30.887652 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:30.888278 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:30.888278 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:30.888348 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:30.888378 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:30.888541 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:30.888546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:30.888550 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:30.888892 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 23:05:30.889012 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260620 23:05:30.889950 7019 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40407
I20260620 23:05:30.890059 7019 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:30.890268 7019 heartbeater.cc:511] Master 127.21.66.126:40407 requested a full tablet report, sending...
I20260620 23:05:30.890878 6831 ts_manager.cc:194] Registered new tserver with Master: 255bcb4e2862483f9a6b07d7b9cbe031 (127.21.66.65:37305)
I20260620 23:05:30.891517 6831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.65:45939
W20260620 23:05:30.974416 7022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:30.974705 7022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:30.974756 7022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:30.976018 7022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:30.976087 7022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:30.976133 7022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:05:30.977626 7022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:30.977911 7022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:30.978253 7022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:30.983350 7028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.983378 7027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:30.983359 7030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:30.983753 7022 server_base.cc:1034] running on GCE node
I20260620 23:05:30.984001 7022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:30.984284 7022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:30.985493 7022 hybrid_clock.cc:648] HybridClock initialized: now 1781996730985472 us; error 24 us; skew 500 ppm
I20260620 23:05:30.986966 7022 webserver.cc:460] Webserver started at http://127.21.66.66:36035/ using document root <none> and password file <none>
I20260620 23:05:30.987242 7022 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:30.987317 7022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:30.987474 7022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:30.988541 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "03bb52e088c14c24adc040e9d7f7cf1f"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "36a3538b780072a07e060a34de7a90e0"
server_key_iv: "348fd65bc1a575e4bb193f4103510af9"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.988914 7022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "03bb52e088c14c24adc040e9d7f7cf1f"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "36a3538b780072a07e060a34de7a90e0"
server_key_iv: "348fd65bc1a575e4bb193f4103510af9"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.990263 7022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:30.991078 7036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:30.991251 7022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:05:30.991354 7022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "03bb52e088c14c24adc040e9d7f7cf1f"
format_stamp: "Formatted at 2026-06-20 23:05:30 on dist-test-slave-qgtk"
server_key: "36a3538b780072a07e060a34de7a90e0"
server_key_iv: "348fd65bc1a575e4bb193f4103510af9"
server_key_version: "encryptionkey@0"
I20260620 23:05:30.991441 7022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:31.010686 7022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:31.011041 7022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:31.011219 7022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:31.011471 7022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:31.011826 7022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:31.011883 7022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.011927 7022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:31.011951 7022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.018071 7022 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:37005
I20260620 23:05:31.018151 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:37005 every 8 connection(s)
I20260620 23:05:31.018519 7022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:31.024274 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7022
I20260620 23:05:31.024421 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:31.024405 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.019372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:48739 (local address 127.21.66.126:40407)
0620 23:05:31.019458 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.019461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.019758 (+ 297us) server_negotiation.cc:407] Connection header received
0620 23:05:31.020265 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.020266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.020278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.020341 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:31.020991 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.021537 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.022812 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.023026 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.023090 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.024108 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:31.024109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:31.024110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:31.024114 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:31.024125 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:31.024129 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:31.024172 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.024266 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.024288 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:05:31.024655 21769 external_mini_cluster.cc:1428] Setting key 1c8979a1522a588a542c201ef450baca
I20260620 23:05:31.024679 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.019355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40407 (local address 127.21.66.66:48739)
0620 23:05:31.019576 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.019612 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.020166 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.020396 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.020400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.020410 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:31.020917 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.020921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.021635 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.021636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.022722 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.022726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.023129 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.023130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.023298 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.023936 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:31.023936 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:31.023970 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:31.023996 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:31.024175 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:31.024180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:31.024184 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.024266 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.024414 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":118,"threads_started":1}
I20260620 23:05:31.025202 7150 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40407
I20260620 23:05:31.025323 7150 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:31.025543 7150 heartbeater.cc:511] Master 127.21.66.126:40407 requested a full tablet report, sending...
I20260620 23:05:31.025975 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40407
--builtin_ntp_servers=127.21.66.84:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:05:31.026029 6830 ts_manager.cc:194] Registered new tserver with Master: 03bb52e088c14c24adc040e9d7f7cf1f (127.21.66.66:37005)
I20260620 23:05:31.026520 6830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.66:48739
W20260620 23:05:31.110752 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:31.110982 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:31.111004 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:31.112155 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:31.112197 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:31.112222 7153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:05:31.113536 7153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:40407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:31.113732 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:31.113991 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:31.119571 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:31.119562 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:31.119626 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:31.119908 7153 server_base.cc:1034] running on GCE node
I20260620 23:05:31.120085 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:31.120317 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:31.121471 7153 hybrid_clock.cc:648] HybridClock initialized: now 1781996731121459 us; error 26 us; skew 500 ppm
I20260620 23:05:31.122877 7153 webserver.cc:460] Webserver started at http://127.21.66.67:46583/ using document root <none> and password file <none>
I20260620 23:05:31.123080 7153 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:31.123122 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:31.123216 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:31.124279 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "27ff8b8e59144da5b17c613b89240906"
format_stamp: "Formatted at 2026-06-20 23:05:31 on dist-test-slave-qgtk"
server_key: "d83769096dba27592d4f0b04514a4be9"
server_key_iv: "628db5a88425ad3da8b5bcee83bf3907"
server_key_version: "encryptionkey@0"
I20260620 23:05:31.124593 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "27ff8b8e59144da5b17c613b89240906"
format_stamp: "Formatted at 2026-06-20 23:05:31 on dist-test-slave-qgtk"
server_key: "d83769096dba27592d4f0b04514a4be9"
server_key_iv: "628db5a88425ad3da8b5bcee83bf3907"
server_key_version: "encryptionkey@0"
I20260620 23:05:31.125802 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:05:31.126573 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.126824 7153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:05:31.126924 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "27ff8b8e59144da5b17c613b89240906"
format_stamp: "Formatted at 2026-06-20 23:05:31 on dist-test-slave-qgtk"
server_key: "d83769096dba27592d4f0b04514a4be9"
server_key_iv: "628db5a88425ad3da8b5bcee83bf3907"
server_key_version: "encryptionkey@0"
I20260620 23:05:31.127012 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:31.160187 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:31.160533 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:31.160692 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:31.160948 7153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:31.161301 7153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:31.161360 7153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.161399 7153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:31.161444 7153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:31.167294 7153 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:46717
I20260620 23:05:31.167423 7280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:46717 every 8 connection(s)
I20260620 23:05:31.167717 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:31.171717 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7153
I20260620 23:05:31.171855 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:31.172080 21769 external_mini_cluster.cc:1428] Setting key f21d432347900d730765212e7b6061c3
I20260620 23:05:31.173454 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.168543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:58725 (local address 127.21.66.126:40407)
0620 23:05:31.168606 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.168608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.168902 (+ 294us) server_negotiation.cc:407] Connection header received
0620 23:05:31.169433 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.169434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.169445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.169504 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:31.170223 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.170827 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.171836 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.172086 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.172170 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.173188 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:31.173190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:31.173190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:31.173195 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:31.173210 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:31.173214 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:31.173268 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.173350 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.173376 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 23:05:31.173712 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.168536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:40407 (local address 127.21.66.67:58725)
0620 23:05:31.168743 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.168778 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.169328 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.169584 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.169588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.169597 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:31.170091 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.170094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.170931 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.170933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.171723 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.171726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.172218 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.172219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.172376 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.173012 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:31.173013 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:31.173053 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:31.173080 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:31.173270 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:31.173276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:31.173279 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.173352 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.173456 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260620 23:05:31.174202 7281 heartbeater.cc:347] Connected to a master server at 127.21.66.126:40407
I20260620 23:05:31.174324 7281 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:31.174533 7281 heartbeater.cc:511] Master 127.21.66.126:40407 requested a full tablet report, sending...
I20260620 23:05:31.175005 6830 ts_manager.cc:194] Registered new tserver with Master: 27ff8b8e59144da5b17c613b89240906 (127.21.66.67:46717)
I20260620 23:05:31.175462 6830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.67:58725
I20260620 23:05:31.184243 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:31.188920 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.185804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.21.66.126:40407)
0620 23:05:31.185893 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.185895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.186050 (+ 155us) server_negotiation.cc:407] Connection header received
0620 23:05:31.186093 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.186094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.186104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.186249 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:31.186410 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.187025 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.187790 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.188002 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.188066 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.188695 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:31.188697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:31.188697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:31.188702 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:31.188717 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:31.188721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:31.188779 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.188832 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.188853 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:05:31.190892 6830 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:31.192504 6830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:31.200034 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.195517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37305 (local address 127.0.0.1:36046)
0620 23:05:31.195613 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.195625 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.195676 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.196303 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.196304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.196306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.196369 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.196371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.197742 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.197743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.198707 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.198710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.199667 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.199669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.199765 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.199782 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.199869 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.199929 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
I20260620 23:05:31.200058 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.195591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.21.66.65:37305)
0620 23:05:31.195820 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.195825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.196082 (+ 257us) server_negotiation.cc:407] Connection header received
0620 23:05:31.196131 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.196134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.196169 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.196303 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.196454 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.197626 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.198823 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.199568 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.199652 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.199753 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.199898 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.199945 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260620 23:05:31.200636 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.195139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:37005 (local address 127.0.0.1:38724)
0620 23:05:31.195351 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.195419 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.195497 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.195888 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.195891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.195895 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.195976 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.195978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.197521 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.197522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.199031 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.199034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.200165 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.200166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.200269 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.200275 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.200486 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.200543 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":26,"thread_start_us":150,"threads_started":1}
I20260620 23:05:31.200831 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.195892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.21.66.67:46717)
0620 23:05:31.196149 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.196154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.196164 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:05:31.196208 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.196211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.196228 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.196321 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.196568 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.197358 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.199305 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.200040 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.200111 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.200330 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.200597 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.200651 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
I20260620 23:05:31.200877 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.195256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.21.66.66:37005)
0620 23:05:31.195456 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.195462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.195492 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:05:31.195676 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.195678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.195697 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.195790 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.196087 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.197444 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.199128 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.199953 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.200027 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.200122 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.200747 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.200795 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260620 23:05:31.201040 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.195674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:46717 (local address 127.0.0.1:42386)
0620 23:05:31.195797 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.195809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.195853 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.196457 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.196458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.196459 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.196486 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.196487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.197461 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.197462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.199216 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.199218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.200366 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.200367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.200435 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.200439 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.200495 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.200546 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":52,"thread_start_us":69,"threads_started":1}
I20260620 23:05:31.200991 6952 tablet_service.cc:1467] Processing CreateTablet for tablet 147f1f455a8f4775a503ebcf5252a362 (DEFAULT_TABLE table=test-table [id=5169c9f10ed744cdaac62fcd23292ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:31.201432 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147f1f455a8f4775a503ebcf5252a362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:31.201699 7215 tablet_service.cc:1467] Processing CreateTablet for tablet 147f1f455a8f4775a503ebcf5252a362 (DEFAULT_TABLE table=test-table [id=5169c9f10ed744cdaac62fcd23292ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:31.201964 7215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147f1f455a8f4775a503ebcf5252a362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:31.202600 7084 tablet_service.cc:1467] Processing CreateTablet for tablet 147f1f455a8f4775a503ebcf5252a362 (DEFAULT_TABLE table=test-table [id=5169c9f10ed744cdaac62fcd23292ffe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:31.202944 7084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147f1f455a8f4775a503ebcf5252a362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:31.204845 7295 tablet_bootstrap.cc:492] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031: Bootstrap starting.
I20260620 23:05:31.205749 7295 tablet_bootstrap.cc:654] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:31.205948 7296 tablet_bootstrap.cc:492] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Bootstrap starting.
I20260620 23:05:31.206224 7295 log.cc:824] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:31.206684 7296 tablet_bootstrap.cc:654] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:31.206970 7298 tablet_bootstrap.cc:492] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: Bootstrap starting.
I20260620 23:05:31.207080 7296 log.cc:824] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:31.207454 7295 tablet_bootstrap.cc:492] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031: No bootstrap required, opened a new log
I20260620 23:05:31.207535 7295 ts_tablet_manager.cc:1397] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 23:05:31.207782 7298 tablet_bootstrap.cc:654] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:31.207899 7296 tablet_bootstrap.cc:492] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: No bootstrap required, opened a new log
I20260620 23:05:31.207974 7296 ts_tablet_manager.cc:1397] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:31.208133 7298 log.cc:824] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:31.209013 7298 tablet_bootstrap.cc:492] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: No bootstrap required, opened a new log
I20260620 23:05:31.209092 7298 ts_tablet_manager.cc:1397] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:31.210351 7296 raft_consensus.cc:348] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.210520 7296 raft_consensus.cc:374] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:31.210568 7296 raft_consensus.cc:729] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ff8b8e59144da5b17c613b89240906, State: Initialized, Role: FOLLOWER
I20260620 23:05:31.210695 7296 consensus_queue.cc:260] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.210632 7295 raft_consensus.cc:348] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.210821 7295 raft_consensus.cc:374] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:31.210870 7295 raft_consensus.cc:729] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255bcb4e2862483f9a6b07d7b9cbe031, State: Initialized, Role: FOLLOWER
I20260620 23:05:31.210996 7296 ts_tablet_manager.cc:1428] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:31.211115 7281 heartbeater.cc:503] Master 127.21.66.126:40407 was elected leader, sending a full tablet report...
I20260620 23:05:31.211402 7295 consensus_queue.cc:260] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.211699 7295 ts_tablet_manager.cc:1428] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260620 23:05:31.211843 7019 heartbeater.cc:503] Master 127.21.66.126:40407 was elected leader, sending a full tablet report...
I20260620 23:05:31.212378 7298 raft_consensus.cc:348] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.212568 7298 raft_consensus.cc:374] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:31.212597 7298 raft_consensus.cc:729] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bb52e088c14c24adc040e9d7f7cf1f, State: Initialized, Role: FOLLOWER
I20260620 23:05:31.212741 7298 consensus_queue.cc:260] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.213025 7298 ts_tablet_manager.cc:1428] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 23:05:31.213109 7150 heartbeater.cc:503] Master 127.21.66.126:40407 was elected leader, sending a full tablet report...
W20260620 23:05:31.269160 7151 tablet.cc:2369] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:31.300088 7302 raft_consensus.cc:482] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:31.300218 7302 raft_consensus.cc:504] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.300710 7302 leader_election.cc:288] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bb52e088c14c24adc040e9d7f7cf1f (127.21.66.66:37005), 27ff8b8e59144da5b17c613b89240906 (127.21.66.67:46717)
I20260620 23:05:31.303589 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.300930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:37005 (local address 127.21.66.65:45879)
0620 23:05:31.300996 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.301011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.301092 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.301249 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.301250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.301252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.301325 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.301327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.301986 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.301988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.302764 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.302766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.303322 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.303323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.303405 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.303407 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.303470 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.303514 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:05:31.303589 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.301047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:45879 (local address 127.21.66.66:37005)
0620 23:05:31.301099 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.301101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.301139 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:05:31.301168 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.301169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.301181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.301232 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.301406 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.301895 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.302882 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.303237 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.303333 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.303343 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.303488 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.303526 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:05:31.303985 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.301071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:46717 (local address 127.21.66.65:59571)
0620 23:05:31.301265 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.301276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:31.301362 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:31.301489 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:31.301489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:31.301491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:31.301564 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.301566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.302342 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.302343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.303079 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:31.303080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.303686 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.303687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:31.303765 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.303780 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 23:05:31.303860 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 23:05:31.303900 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":118,"threads_started":1}
I20260620 23:05:31.303970 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.301174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:59571 (local address 127.21.66.67:46717)
0620 23:05:31.301239 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.301241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.301353 (+ 112us) server_negotiation.cc:407] Connection header received
0620 23:05:31.301422 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.301423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.301433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.301478 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:31.301633 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.302247 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.303170 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.303597 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.303680 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.303687 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.303872 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.303907 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:05:31.303929 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147f1f455a8f4775a503ebcf5252a362" candidate_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" is_pre_election: true
I20260620 23:05:31.304147 7104 raft_consensus.cc:2454] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 255bcb4e2862483f9a6b07d7b9cbe031 in term 0.
I20260620 23:05:31.304410 6907 leader_election.cc:302] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bb52e088c14c24adc040e9d7f7cf1f, 255bcb4e2862483f9a6b07d7b9cbe031; no voters:
I20260620 23:05:31.304575 7302 raft_consensus.cc:2781] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:31.304627 7302 raft_consensus.cc:482] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:31.304661 7302 raft_consensus.cc:3037] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:31.304587 7235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147f1f455a8f4775a503ebcf5252a362" candidate_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ff8b8e59144da5b17c613b89240906" is_pre_election: true
I20260620 23:05:31.304759 7235 raft_consensus.cc:2454] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 255bcb4e2862483f9a6b07d7b9cbe031 in term 0.
I20260620 23:05:31.305510 7302 raft_consensus.cc:504] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.305653 7302 leader_election.cc:288] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [CANDIDATE]: Term 1 election: Requested vote from peers 03bb52e088c14c24adc040e9d7f7cf1f (127.21.66.66:37005), 27ff8b8e59144da5b17c613b89240906 (127.21.66.67:46717)
I20260620 23:05:31.305938 7235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147f1f455a8f4775a503ebcf5252a362" candidate_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ff8b8e59144da5b17c613b89240906"
I20260620 23:05:31.305948 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147f1f455a8f4775a503ebcf5252a362" candidate_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bb52e088c14c24adc040e9d7f7cf1f"
I20260620 23:05:31.306021 7235 raft_consensus.cc:3037] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:31.306021 7104 raft_consensus.cc:3037] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:31.306943 7235 raft_consensus.cc:2454] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 255bcb4e2862483f9a6b07d7b9cbe031 in term 1.
I20260620 23:05:31.306941 7104 raft_consensus.cc:2454] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 255bcb4e2862483f9a6b07d7b9cbe031 in term 1.
I20260620 23:05:31.307116 6907 leader_election.cc:302] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03bb52e088c14c24adc040e9d7f7cf1f, 255bcb4e2862483f9a6b07d7b9cbe031; no voters:
I20260620 23:05:31.307229 7302 raft_consensus.cc:2781] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:31.307375 7302 raft_consensus.cc:686] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 1 LEADER]: Becoming Leader. State: Replica: 255bcb4e2862483f9a6b07d7b9cbe031, State: Running, Role: LEADER
I20260620 23:05:31.307483 7302 consensus_queue.cc:237] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } }
I20260620 23:05:31.308403 6831 catalog_manager.cc:5495] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 255bcb4e2862483f9a6b07d7b9cbe031 (127.21.66.65). New cstate: current_term: 1 leader_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "255bcb4e2862483f9a6b07d7b9cbe031" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 37305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:31.316284 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.313378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.21.66.65:37305)
0620 23:05:31.313420 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:31.313422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:31.313437 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:05:31.313527 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:31.313528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:31.313550 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:31.313628 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:31.313796 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.314626 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.315433 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:31.315617 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:31.315670 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:31.315818 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:31.315920 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:31.315984 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:31.316030 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:05:31.316048 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:05:31.318243 7104 raft_consensus.cc:1261] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 1 FOLLOWER]: Refusing update from remote peer 255bcb4e2862483f9a6b07d7b9cbe031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:31.318248 7235 raft_consensus.cc:1261] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 1 FOLLOWER]: Refusing update from remote peer 255bcb4e2862483f9a6b07d7b9cbe031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:31.318612 7305 consensus_queue.cc:1035] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ff8b8e59144da5b17c613b89240906" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:31.318749 7302 consensus_queue.cc:1035] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bb52e088c14c24adc040e9d7f7cf1f" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:31.322619 7309 mvcc.cc:205] Tried to move back new op lower bound from 7299058611476455424 to 7299058611436126208. Current Snapshot: MvccSnapshot[applied={T|T < 7299058611476455424}]
I20260620 23:05:31.322746 7310 mvcc.cc:205] Tried to move back new op lower bound from 7299058611476455424 to 7299058611436126208. Current Snapshot: MvccSnapshot[applied={T|T < 7299058611476455424 or (T in {7299058611476455424})}]
I20260620 23:05:31.323635 6831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:31.323733 6831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:31.324544 6831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e159b6bb05364fd084ae88b5689506a0: Sending DeleteTablet for 3 replicas of tablet 147f1f455a8f4775a503ebcf5252a362
I20260620 23:05:31.324884 7084 tablet_service.cc:1514] Processing DeleteTablet for tablet 147f1f455a8f4775a503ebcf5252a362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:31 UTC) from {username='slave'} at 127.0.0.1:38724
I20260620 23:05:31.324914 6952 tablet_service.cc:1514] Processing DeleteTablet for tablet 147f1f455a8f4775a503ebcf5252a362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:31 UTC) from {username='slave'} at 127.0.0.1:36046
I20260620 23:05:31.324972 7215 tablet_service.cc:1514] Processing DeleteTablet for tablet 147f1f455a8f4775a503ebcf5252a362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:31 UTC) from {username='slave'} at 127.0.0.1:42386
I20260620 23:05:31.325140 7319 tablet_replica.cc:330] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031: stopping tablet replica
I20260620 23:05:31.325305 7319 raft_consensus.cc:2229] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:31.325416 7319 raft_consensus.cc:2258] T 147f1f455a8f4775a503ebcf5252a362 P 255bcb4e2862483f9a6b07d7b9cbe031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:31.325555 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 6891
I20260620 23:05:31.325577 7320 tablet_replica.cc:330] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: stopping tablet replica
I20260620 23:05:31.325664 7320 raft_consensus.cc:2229] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:31.325763 7320 raft_consensus.cc:2258] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:31.326356 7320 ts_tablet_manager.cc:1905] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:31.328341 7320 ts_tablet_manager.cc:1918] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:31.328418 7320 log.cc:1196] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/147f1f455a8f4775a503ebcf5252a362
I20260620 23:05:31.328647 7320 ts_tablet_manager.cc:1939] T 147f1f455a8f4775a503ebcf5252a362 P 27ff8b8e59144da5b17c613b89240906: Deleting consensus metadata
I20260620 23:05:31.329200 6816 catalog_manager.cc:4841] TS 27ff8b8e59144da5b17c613b89240906 (127.21.66.67:46717): tablet 147f1f455a8f4775a503ebcf5252a362 (table test-table [id=5169c9f10ed744cdaac62fcd23292ffe]) successfully deleted
W20260620 23:05:31.332510 6816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:37305 (error 108)
I20260620 23:05:31.332979 7318 tablet_replica.cc:330] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f: stopping tablet replica
I20260620 23:05:31.333082 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.332928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:37305 (local address 127.0.0.1:36062)
0620 23:05:31.332984 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.333025 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:37305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 23:05:31.333112 7318 raft_consensus.cc:2229] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 23:05:31.333190 6816 catalog_manager.cc:4584] TS 255bcb4e2862483f9a6b07d7b9cbe031 (127.21.66.65:37305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 147f1f455a8f4775a503ebcf5252a362: Network error: Client connection negotiation failed: client connection to 127.21.66.65:37305: connect: Connection refused (error 111)
I20260620 23:05:31.333248 7318 raft_consensus.cc:2258] T 147f1f455a8f4775a503ebcf5252a362 P 03bb52e088c14c24adc040e9d7f7cf1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:31.333472 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7022
I20260620 23:05:31.338609 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7153
I20260620 23:05:31.338639 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:31.338509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:37005 (local address 127.0.0.1:38740)
0620 23:05:31.338557 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:31.338592 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.66:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 23:05:31.343297 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 6800
2026-06-20T23:05:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (796 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:31 dist-test-slave-qgtk krb5kdc[7324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:31 dist-test-slave-qgtk krb5kdc[7324](info): set up 2 sockets
Jun 20 23:05:31 dist-test-slave-qgtk krb5kdc[7324](info): commencing operation
krb5kdc: starting...
W20260620 23:05:32.449954 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:32 dist-test-slave-qgtk krb5kdc[7324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:05:32.577309 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45169
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:45169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:32.660624 7337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:32.660852 7337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:32.660876 7337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:32.661995 7337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:32.662035 7337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:32.662050 7337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:32.662063 7337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:32.662076 7337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:32.663606 7337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:45169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:45169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:32.663856 7337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:32.664068 7337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:32.669628 7343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:32.669711 7337 server_base.cc:1034] running on GCE node
W20260620 23:05:32.669631 7342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:32.669658 7345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:32.670195 7337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:32.670547 7337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:32.671774 7337 hybrid_clock.cc:648] HybridClock initialized: now 1781996732671762 us; error 30 us; skew 500 ppm
Jun 20 23:05:32 dist-test-slave-qgtk krb5kdc[7324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996732, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:32.674639 7337 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:32.675124 7337 webserver.cc:460] Webserver started at http://127.21.66.126:43415/ using document root <none> and password file <none>
I20260620 23:05:32.675333 7337 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:32.675374 7337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:32.675467 7337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:32.676492 7337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "b8983e721f02400b8d50ff89d4c34167"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-qgtk"
server_key: "465f8bbf838cee9cb8de309881b402ab"
server_key_iv: "864ec06effaee848a6496ac9331f0ffd"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.676801 7337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "b8983e721f02400b8d50ff89d4c34167"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-qgtk"
server_key: "465f8bbf838cee9cb8de309881b402ab"
server_key_iv: "864ec06effaee848a6496ac9331f0ffd"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.678213 7337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:32.679028 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:32.679214 7337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:32.679311 7337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "b8983e721f02400b8d50ff89d4c34167"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-qgtk"
server_key: "465f8bbf838cee9cb8de309881b402ab"
server_key_iv: "864ec06effaee848a6496ac9331f0ffd"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.679374 7337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:32.696877 7337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:32.699743 7337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:32.699874 7337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:32.704272 7337 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:45169
I20260620 23:05:32.704370 7404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:45169 every 8 connection(s)
I20260620 23:05:32.704751 7337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:32.705704 7405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:32.708115 7405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167: Bootstrap starting.
I20260620 23:05:32.708715 7405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:32.709013 7405 log.cc:824] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:32.709779 7405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167: No bootstrap required, opened a new log
I20260620 23:05:32.711457 7405 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8983e721f02400b8d50ff89d4c34167" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45169 } }
I20260620 23:05:32.711625 7405 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:32.711676 7405 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8983e721f02400b8d50ff89d4c34167, State: Initialized, Role: FOLLOWER
I20260620 23:05:32.711827 7405 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8983e721f02400b8d50ff89d4c34167" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45169 } }
I20260620 23:05:32.711908 7405 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:32.711952 7405 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:32.712008 7405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:32.712533 7405 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8983e721f02400b8d50ff89d4c34167" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45169 } }
I20260620 23:05:32.712653 7405 leader_election.cc:302] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8983e721f02400b8d50ff89d4c34167; no voters:
I20260620 23:05:32.712848 7405 leader_election.cc:288] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:32.712855 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7337
I20260620 23:05:32.712909 7408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:32.712975 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:32.713075 7408 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [term 1 LEADER]: Becoming Leader. State: Replica: b8983e721f02400b8d50ff89d4c34167, State: Running, Role: LEADER
I20260620 23:05:32.713136 7405 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:32.713183 21769 external_mini_cluster.cc:1428] Setting key 6c75a195a9a6c4b692f41ab2ab9e2881
I20260620 23:05:32.713166 7408 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8983e721f02400b8d50ff89d4c34167" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45169 } }
I20260620 23:05:32.713483 7408 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8983e721f02400b8d50ff89d4c34167. Latest consensus state: current_term: 1 leader_uuid: "b8983e721f02400b8d50ff89d4c34167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8983e721f02400b8d50ff89d4c34167" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45169 } } }
I20260620 23:05:32.713562 7408 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:32.713657 7409 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8983e721f02400b8d50ff89d4c34167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8983e721f02400b8d50ff89d4c34167" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 45169 } } }
I20260620 23:05:32.713738 7409 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:32.714286 7413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:32.714795 7413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:32.716495 7413 catalog_manager.cc:1347] Generated new cluster ID: 0eba3bdeebdc4d5ea181a28b04efa4fa
I20260620 23:05:32.716547 7413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:05:32 dist-test-slave-qgtk krb5kdc[7324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:32.725703 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:32.713763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.21.66.126:45169)
0620 23:05:32.714189 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:32.714200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:32.714238 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:05:32.714404 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:32.714411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:32.714483 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:32.715616 (+ 1133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:32.715688 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.716977 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.718090 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.718810 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.718911 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:32.721773 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:32.721778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:32.721778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:32.721792 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:32.723923 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.724332 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.724336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.724336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.724468 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.724675 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.724676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.724676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.725042 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:32.725197 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:32.725326 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 23:05:32.725476 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":71,"threads_started":1}
I20260620 23:05:32.726423 7413 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:32.726944 7413 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:32.732563 7413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167: Generated new TSK 0
I20260620 23:05:32.732748 7413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:05:32.760674 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45169
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:32.844154 7431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:32.844415 7431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:32.844452 7431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:32.845669 7431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:32.845749 7431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:32.845793 7431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:05:32.847328 7431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:32.847625 7431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:32.847874 7431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:32.853490 7439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:32.853519 7437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:32.853637 7436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:32.853838 7431 server_base.cc:1034] running on GCE node
I20260620 23:05:32.854072 7431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:32.854444 7431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:32.855651 7431 hybrid_clock.cc:648] HybridClock initialized: now 1781996732855631 us; error 39 us; skew 500 ppm
Jun 20 23:05:32 dist-test-slave-qgtk krb5kdc[7324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996732, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:32.858420 7431 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:05:32.858933 7431 webserver.cc:460] Webserver started at http://127.21.66.65:39053/ using document root <none> and password file <none>
I20260620 23:05:32.859208 7431 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:32.859285 7431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:32.859453 7431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:32.860605 7431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "23b259fb919b421eb10d99e6603783c2"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-qgtk"
server_key: "65c309643f2ad61ff65c659d1a707612"
server_key_iv: "096afbb126ae99297868c6e3eee9003b"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.860985 7431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "23b259fb919b421eb10d99e6603783c2"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-qgtk"
server_key: "65c309643f2ad61ff65c659d1a707612"
server_key_iv: "096afbb126ae99297868c6e3eee9003b"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.862592 7431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:32.863669 7446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:32.863871 7431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:32.863986 7431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "23b259fb919b421eb10d99e6603783c2"
format_stamp: "Formatted at 2026-06-20 23:05:32 on dist-test-slave-qgtk"
server_key: "65c309643f2ad61ff65c659d1a707612"
server_key_iv: "096afbb126ae99297868c6e3eee9003b"
server_key_version: "encryptionkey@0"
I20260620 23:05:32.864107 7431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:32.875161 7431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:32.877384 7431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:32.877562 7431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:32.877825 7431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:32.878189 7431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:32.878289 7431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:32.878342 7431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:32.878368 7431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:32.884784 7431 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:45225
I20260620 23:05:32.884832 7559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:45225 every 8 connection(s)
I20260620 23:05:32.885180 7431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:32.886297 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7431
I20260620 23:05:32.886435 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:32.886650 21769 external_mini_cluster.cc:1428] Setting key 4fe9234e1500fc35dc764fb7305a5c38
Jun 20 23:05:32 dist-test-slave-qgtk krb5kdc[7324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996732, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
I20260620 23:05:32.896992 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:32.885936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:57679 (local address 127.21.66.126:45169)
0620 23:05:32.886014 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:32.886043 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:32.886341 (+ 298us) server_negotiation.cc:407] Connection header received
0620 23:05:32.887077 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:32.887078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:32.887093 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:32.887140 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:32.888602 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.889127 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.890742 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.890962 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.891062 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:32.893721 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:32.893723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:32.893723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:32.893731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:32.895403 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.895902 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.895904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.895904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.895971 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:32.896264 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:32.896265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:32.896266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:32.896395 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:32.896480 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:32.896792 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 23:05:32.896888 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:05:32.897239 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:32.885924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45169 (local address 127.21.66.65:57679)
0620 23:05:32.886186 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:32.886247 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:32.886953 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:32.887222 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:32.887226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:32.888189 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:32.888494 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:32.888500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.889228 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.889230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:32.890635 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:32.890639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:32.891060 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:32.891061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:32.891268 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:32.891851 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:32.891863 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:32.893611 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:32.895536 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:32.895544 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:32.895546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:32.895801 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:32.896063 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:32.896064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:32.896064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:32.896187 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:32.896471 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:32.896478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:32.896643 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 23:05:32.896768 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:05:32.896951 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
I20260620 23:05:32.897863 7560 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45169
I20260620 23:05:32.897981 7560 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:32.898216 7560 heartbeater.cc:511] Master 127.21.66.126:45169 requested a full tablet report, sending...
I20260620 23:05:32.898962 7369 ts_manager.cc:194] Registered new tserver with Master: 23b259fb919b421eb10d99e6603783c2 (127.21.66.65:45225)
I20260620 23:05:32.899969 7369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:57679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:05:32.913241 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45169
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:33.006034 7565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:33.006304 7565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:33.006332 7565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:33.007681 7565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:33.007730 7565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:33.007758 7565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:05:33.009518 7565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:33.009799 7565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:33.010069 7565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:33.015740 7570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.015789 7573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.015792 7571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:33.016158 7565 server_base.cc:1034] running on GCE node
I20260620 23:05:33.016424 7565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:33.016692 7565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:33.017868 7565 hybrid_clock.cc:648] HybridClock initialized: now 1781996733017865 us; error 40 us; skew 500 ppm
Jun 20 23:05:33 dist-test-slave-qgtk krb5kdc[7324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:33.020627 7565 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:05:33.021132 7565 webserver.cc:460] Webserver started at http://127.21.66.66:34613/ using document root <none> and password file <none>
I20260620 23:05:33.021384 7565 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:33.021452 7565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:33.021611 7565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:33.022773 7565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "54cde0193bec4dd487bcdc7a34688d17"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "faf82d9576818ca7f9482792eea7e51d"
server_key_iv: "614d9dd309f12e557b243488c3d88c29"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.023137 7565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "54cde0193bec4dd487bcdc7a34688d17"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "faf82d9576818ca7f9482792eea7e51d"
server_key_iv: "614d9dd309f12e557b243488c3d88c29"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.024592 7565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:05:33.025578 7580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.025851 7565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:33.026019 7565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "54cde0193bec4dd487bcdc7a34688d17"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "faf82d9576818ca7f9482792eea7e51d"
server_key_iv: "614d9dd309f12e557b243488c3d88c29"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.026113 7565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:33.037318 7565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:33.039861 7565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:33.040064 7565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:33.040365 7565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:33.040719 7565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:33.040779 7565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.040823 7565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:33.040851 7565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.047538 7565 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:46273
I20260620 23:05:33.047586 7693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:46273 every 8 connection(s)
I20260620 23:05:33.047961 7565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:33.048810 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7565
I20260620 23:05:33.048934 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:33.049162 21769 external_mini_cluster.cc:1428] Setting key d0d207bf5caba68dd3620db8c48dcf37
Jun 20 23:05:33 dist-test-slave-qgtk krb5kdc[7324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
I20260620 23:05:33.060214 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.048797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:32923 (local address 127.21.66.126:45169)
0620 23:05:33.048884 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.048887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.049129 (+ 242us) server_negotiation.cc:407] Connection header received
0620 23:05:33.049852 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.049853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.049866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.049916 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:33.051037 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.051625 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.053357 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.053650 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.053761 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.056668 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:33.056670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:33.056670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:33.056680 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:33.058595 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:33.059052 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:33.059054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:33.059054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:33.059115 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:33.059444 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:33.059445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:33.059446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:33.059579 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:33.059650 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.059990 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.060113 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 23:05:33.060416 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.048761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45169 (local address 127.21.66.66:32923)
0620 23:05:33.048973 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.049020 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.049734 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.050002 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.050007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.050687 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:33.050903 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.050909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.051738 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.051740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.053231 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.053235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.053820 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.053821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.054004 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.054656 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:33.054668 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:33.056531 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:33.058694 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:33.058703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:33.058705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:33.058944 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:33.059244 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:33.059245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:33.059246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:33.059362 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:33.059652 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:33.059660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:33.059832 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.059970 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.060177 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
I20260620 23:05:33.061085 7694 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45169
I20260620 23:05:33.061208 7694 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:33.061522 7694 heartbeater.cc:511] Master 127.21.66.126:45169 requested a full tablet report, sending...
I20260620 23:05:33.062002 7369 ts_manager.cc:194] Registered new tserver with Master: 54cde0193bec4dd487bcdc7a34688d17 (127.21.66.66:46273)
I20260620 23:05:33.062628 7369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:32923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:05:33.077296 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45169
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:33.169212 7699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:33.169461 7699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:33.169489 7699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:33.170950 7699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:33.171006 7699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:33.171037 7699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:05:33.172657 7699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:45169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:33.172865 7699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:33.173065 7699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:33.178774 7705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.178748 7707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.178748 7704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:33.179139 7699 server_base.cc:1034] running on GCE node
I20260620 23:05:33.179355 7699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:33.179593 7699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:33.180755 7699 hybrid_clock.cc:648] HybridClock initialized: now 1781996733180747 us; error 36 us; skew 500 ppm
Jun 20 23:05:33 dist-test-slave-qgtk krb5kdc[7324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:33.192169 7699 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:05:33.192780 7699 webserver.cc:460] Webserver started at http://127.21.66.67:46707/ using document root <none> and password file <none>
I20260620 23:05:33.193053 7699 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:33.193106 7699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:33.193267 7699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:33.194576 7699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "3cdf2903aa9c435f817b2403c7bf1a54"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "ccfbcc3dd0527c7b4bf454163cb251dd"
server_key_iv: "a9961b7cb7fbc4243c146e1eb34698bb"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.194993 7699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "3cdf2903aa9c435f817b2403c7bf1a54"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "ccfbcc3dd0527c7b4bf454163cb251dd"
server_key_iv: "a9961b7cb7fbc4243c146e1eb34698bb"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.196491 7699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:05:33.197435 7714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.197702 7699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:33.197783 7699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "3cdf2903aa9c435f817b2403c7bf1a54"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "ccfbcc3dd0527c7b4bf454163cb251dd"
server_key_iv: "a9961b7cb7fbc4243c146e1eb34698bb"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.197883 7699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:33.217384 7699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:33.220129 7699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:33.220306 7699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:33.220589 7699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:33.220966 7699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:33.221027 7699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.221076 7699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:33.221107 7699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.230031 7699 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:37557
I20260620 23:05:33.230101 7827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:37557 every 8 connection(s)
I20260620 23:05:33.232689 7699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:33.233330 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7699
I20260620 23:05:33.233443 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:33.233768 21769 external_mini_cluster.cc:1428] Setting key e6d1e617fa78565161de7e3c16987bf7
Jun 20 23:05:33 dist-test-slave-qgtk krb5kdc[7324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:33.248091 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.234971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:59545 (local address 127.21.66.126:45169)
0620 23:05:33.235022 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.235025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.235034 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:05:33.235704 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.235705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.235716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.235749 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:33.237273 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.238348 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.240053 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.240308 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.240369 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.243157 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:33.243158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:33.243158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:33.243166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:33.246104 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:33.246731 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:33.246732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:33.246732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:33.246810 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:33.247141 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:33.247142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:33.247142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:33.247264 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:33.247322 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.247765 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.247880 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:05:33.248258 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.234657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:45169 (local address 127.21.66.67:59545)
0620 23:05:33.234860 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.234935 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.235577 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.235943 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.235948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.236780 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:33.237140 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.237147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.238466 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.238467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.239950 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.239953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.240396 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.240397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.240552 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.241106 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:33.241118 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:33.243037 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:33.246233 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:33.246243 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:33.246247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:33.246641 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:33.246902 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:33.246904 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:33.246904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:33.247067 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:33.247338 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:33.247344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:33.247540 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.247710 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.247965 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
I20260620 23:05:33.248867 7828 heartbeater.cc:347] Connected to a master server at 127.21.66.126:45169
I20260620 23:05:33.248998 7828 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:33.249228 7828 heartbeater.cc:511] Master 127.21.66.126:45169 requested a full tablet report, sending...
I20260620 23:05:33.249657 7369 ts_manager.cc:194] Registered new tserver with Master: 3cdf2903aa9c435f817b2403c7bf1a54 (127.21.66.67:37557)
I20260620 23:05:33.250250 7369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:59545
I20260620 23:05:33.256472 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:33.268183 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.257950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.21.66.126:45169)
0620 23:05:33.258009 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.258011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.258144 (+ 133us) server_negotiation.cc:407] Connection header received
0620 23:05:33.258264 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.258265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.258279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.258367 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:33.259402 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.260273 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.261573 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.262177 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.262257 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.263791 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:33.263792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:33.263793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:33.263800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:33.265874 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:33.266774 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:33.266775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:33.266776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:33.266853 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:33.267172 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:33.267174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:33.267174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:33.267425 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:33.267814 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.267941 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.268014 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:05:33.270314 7369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:33.272964 7369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:33.280419 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.275475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:46273 (local address 127.0.0.1:52682)
0620 23:05:33.275613 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.275659 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.275724 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.276493 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.276497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.276500 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:33.276576 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.276579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.277562 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.277563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.278885 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.278895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.280027 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.280028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.280137 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.280156 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.280255 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.280316 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260620 23:05:33.280416 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.275988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.21.66.66:46273)
0620 23:05:33.276189 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.276194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.276206 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:05:33.276251 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.276254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.276271 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.276396 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:33.276668 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.277449 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.278987 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.279823 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.279908 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.279983 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.280276 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.280321 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260620 23:05:33.281509 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.275677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:37557 (local address 127.0.0.1:40190)
0620 23:05:33.275850 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.275860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.275915 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.276575 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.276576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.276578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:33.276670 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.276672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.278038 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.278039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.280330 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.280333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.281289 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.281290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.281354 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.281358 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.281389 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.281430 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1174,"thread_start_us":74,"threads_started":1}
I20260620 23:05:33.281653 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.275475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:45225 (local address 127.0.0.1:50674)
0620 23:05:33.275812 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.275823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.275869 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.276575 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.276576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.276581 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:33.276671 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.276672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.278047 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.278048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.279330 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.279333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.281383 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.281384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.281455 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.281460 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.281527 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.281578 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":81,"thread_start_us":59,"threads_started":1}
I20260620 23:05:33.281551 7628 tablet_service.cc:1467] Processing CreateTablet for tablet 68ee8c0c0cbb4644b2b1c6d34d706311 (DEFAULT_TABLE table=test-table [id=decaec500ba4446d9df1422a3b538b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:33.281919 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.276183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.21.66.67:37557)
0620 23:05:33.276393 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.276398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.276409 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:05:33.276450 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.276452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.276467 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.276747 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:33.276833 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.277923 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.280410 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.281184 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.281693 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.281765 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.281794 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.281845 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260620 23:05:33.281953 7628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ee8c0c0cbb4644b2b1c6d34d706311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:33.282626 7744 tablet_service.cc:1467] Processing CreateTablet for tablet 68ee8c0c0cbb4644b2b1c6d34d706311 (DEFAULT_TABLE table=test-table [id=decaec500ba4446d9df1422a3b538b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:33.282975 7744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ee8c0c0cbb4644b2b1c6d34d706311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:33.285449 7845 tablet_bootstrap.cc:492] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Bootstrap starting.
I20260620 23:05:33.286161 7845 tablet_bootstrap.cc:654] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:33.286774 7846 tablet_bootstrap.cc:492] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Bootstrap starting.
I20260620 23:05:33.286782 7845 log.cc:824] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:33.286837 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.275668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.21.66.65:45225)
0620 23:05:33.276393 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.276398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.276409 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:05:33.276450 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.276452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.276467 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.276746 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:33.276828 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.277923 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.280437 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.281279 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.281349 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.281423 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.281541 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.281891 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":82,"threads_started":1}
I20260620 23:05:33.287490 7494 tablet_service.cc:1467] Processing CreateTablet for tablet 68ee8c0c0cbb4644b2b1c6d34d706311 (DEFAULT_TABLE table=test-table [id=decaec500ba4446d9df1422a3b538b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:33.287801 7494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ee8c0c0cbb4644b2b1c6d34d706311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:33.288059 7846 tablet_bootstrap.cc:654] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:33.288853 7846 log.cc:824] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:33.289719 7846 tablet_bootstrap.cc:492] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: No bootstrap required, opened a new log
I20260620 23:05:33.289814 7846 ts_tablet_manager.cc:1397] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:05:33.290897 7849 tablet_bootstrap.cc:492] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: Bootstrap starting.
I20260620 23:05:33.291539 7849 tablet_bootstrap.cc:654] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:33.292207 7849 log.cc:824] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:33.292450 7846 raft_consensus.cc:348] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.292677 7846 raft_consensus.cc:374] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:33.292738 7846 raft_consensus.cc:729] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdf2903aa9c435f817b2403c7bf1a54, State: Initialized, Role: FOLLOWER
I20260620 23:05:33.292887 7846 consensus_queue.cc:260] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.293263 7849 tablet_bootstrap.cc:492] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: No bootstrap required, opened a new log
I20260620 23:05:33.293247 7828 heartbeater.cc:503] Master 127.21.66.126:45169 was elected leader, sending a full tablet report...
I20260620 23:05:33.293342 7849 ts_tablet_manager.cc:1397] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:33.293248 7846 ts_tablet_manager.cc:1428] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 23:05:33.293665 7845 tablet_bootstrap.cc:492] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: No bootstrap required, opened a new log
I20260620 23:05:33.293754 7845 ts_tablet_manager.cc:1397] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260620 23:05:33.295683 7849 raft_consensus.cc:348] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.295878 7849 raft_consensus.cc:374] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:33.295920 7849 raft_consensus.cc:729] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b259fb919b421eb10d99e6603783c2, State: Initialized, Role: FOLLOWER
I20260620 23:05:33.296039 7849 consensus_queue.cc:260] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.296329 7560 heartbeater.cc:503] Master 127.21.66.126:45169 was elected leader, sending a full tablet report...
I20260620 23:05:33.296339 7849 ts_tablet_manager.cc:1428] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:33.296659 7845 raft_consensus.cc:348] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.296934 7845 raft_consensus.cc:374] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:33.296988 7845 raft_consensus.cc:729] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cde0193bec4dd487bcdc7a34688d17, State: Initialized, Role: FOLLOWER
I20260620 23:05:33.297122 7845 consensus_queue.cc:260] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.297451 7845 ts_tablet_manager.cc:1428] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 23:05:33.297441 7694 heartbeater.cc:503] Master 127.21.66.126:45169 was elected leader, sending a full tablet report...
W20260620 23:05:33.298496 7695 tablet.cc:2369] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:33.386071 7561 tablet.cc:2369] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:05:33.484277 7829 tablet.cc:2369] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:33.496636 7851 raft_consensus.cc:482] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:33.496809 7851 raft_consensus.cc:504] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.497313 7851 leader_election.cc:288] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54cde0193bec4dd487bcdc7a34688d17 (127.21.66.66:46273), 23b259fb919b421eb10d99e6603783c2 (127.21.66.65:45225)
I20260620 23:05:33.500598 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.497595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:50103 (local address 127.21.66.66:46273)
0620 23:05:33.497658 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.497660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.497671 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:05:33.497765 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.497766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.497780 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.497830 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:33.498045 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.498587 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.499557 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.500160 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.500222 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.500231 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.500465 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.500507 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:05:33.500629 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.497540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:46273 (local address 127.21.66.67:50103)
0620 23:05:33.497605 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.497625 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.497674 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.497836 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.497837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.497839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:33.497956 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.497958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.498667 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.498668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.499434 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.499436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.500251 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.500252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.500366 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.500371 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.500458 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.500521 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:05:33.500942 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.497626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:43313 (local address 127.21.66.65:45225)
0620 23:05:33.497728 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.497730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.497876 (+ 146us) server_negotiation.cc:407] Connection header received
0620 23:05:33.497966 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.497968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.497984 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.498056 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:33.498231 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.498730 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.499954 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.500511 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.500577 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.500585 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.500834 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.500879 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:05:33.501163 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.497543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:45225 (local address 127.21.66.67:43313)
0620 23:05:33.497788 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.497799 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.497894 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.498058 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.498059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.498061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:33.498171 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.498173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.498828 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.498830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.499861 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.499864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.500666 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.500667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.500749 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.500753 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.501035 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.501085 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":23,"thread_start_us":101,"threads_started":1}
I20260620 23:05:33.501197 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee8c0c0cbb4644b2b1c6d34d706311" candidate_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54cde0193bec4dd487bcdc7a34688d17" is_pre_election: true
I20260620 23:05:33.501412 7648 raft_consensus.cc:2454] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cdf2903aa9c435f817b2403c7bf1a54 in term 0.
I20260620 23:05:33.501559 7514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee8c0c0cbb4644b2b1c6d34d706311" candidate_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b259fb919b421eb10d99e6603783c2" is_pre_election: true
I20260620 23:05:33.501721 7715 leader_election.cc:302] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cdf2903aa9c435f817b2403c7bf1a54, 54cde0193bec4dd487bcdc7a34688d17; no voters:
I20260620 23:05:33.501742 7514 raft_consensus.cc:2454] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cdf2903aa9c435f817b2403c7bf1a54 in term 0.
I20260620 23:05:33.501981 7851 raft_consensus.cc:2781] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:33.502038 7851 raft_consensus.cc:482] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:33.502055 7851 raft_consensus.cc:3037] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:33.502894 7851 raft_consensus.cc:504] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.503059 7851 leader_election.cc:288] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [CANDIDATE]: Term 1 election: Requested vote from peers 54cde0193bec4dd487bcdc7a34688d17 (127.21.66.66:46273), 23b259fb919b421eb10d99e6603783c2 (127.21.66.65:45225)
I20260620 23:05:33.503271 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee8c0c0cbb4644b2b1c6d34d706311" candidate_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54cde0193bec4dd487bcdc7a34688d17"
I20260620 23:05:33.503289 7514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ee8c0c0cbb4644b2b1c6d34d706311" candidate_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b259fb919b421eb10d99e6603783c2"
I20260620 23:05:33.503366 7648 raft_consensus.cc:3037] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:33.503414 7514 raft_consensus.cc:3037] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:33.504168 7514 raft_consensus.cc:2454] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cdf2903aa9c435f817b2403c7bf1a54 in term 1.
I20260620 23:05:33.504225 7648 raft_consensus.cc:2454] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cdf2903aa9c435f817b2403c7bf1a54 in term 1.
I20260620 23:05:33.504371 7717 leader_election.cc:302] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b259fb919b421eb10d99e6603783c2, 3cdf2903aa9c435f817b2403c7bf1a54; no voters:
I20260620 23:05:33.504491 7851 raft_consensus.cc:2781] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:33.504665 7851 raft_consensus.cc:686] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 1 LEADER]: Becoming Leader. State: Replica: 3cdf2903aa9c435f817b2403c7bf1a54, State: Running, Role: LEADER
I20260620 23:05:33.504781 7851 consensus_queue.cc:237] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } }
I20260620 23:05:33.505646 7368 catalog_manager.cc:5495] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cdf2903aa9c435f817b2403c7bf1a54 (127.21.66.67). New cstate: current_term: 1 leader_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cdf2903aa9c435f817b2403c7bf1a54" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 37557 } health_report { overall_health: HEALTHY } } }
I20260620 23:05:33.566707 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.564081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.21.66.67:37557)
0620 23:05:33.564165 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.564167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.564178 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:05:33.564286 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.564287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.564315 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.564396 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:33.564596 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.565079 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.565833 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.566067 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.566181 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.566346 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:33.566445 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:33.566557 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.566587 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.566600 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:05:33.568756 7648 raft_consensus.cc:1261] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 1 FOLLOWER]: Refusing update from remote peer 3cdf2903aa9c435f817b2403c7bf1a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:33.569145 7514 raft_consensus.cc:1261] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 1 FOLLOWER]: Refusing update from remote peer 3cdf2903aa9c435f817b2403c7bf1a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:33.569164 7855 consensus_queue.cc:1035] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54cde0193bec4dd487bcdc7a34688d17" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:33.569391 7855 consensus_queue.cc:1035] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b259fb919b421eb10d99e6603783c2" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 45225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:33.572214 7859 mvcc.cc:205] Tried to move back new op lower bound from 7299058620694953984 to 7299058620436586496. Current Snapshot: MvccSnapshot[applied={T|T < 7299058620694953984 or (T in {7299058620694953984})}]
I20260620 23:05:33.572417 7858 mvcc.cc:205] Tried to move back new op lower bound from 7299058620694953984 to 7299058620436586496. Current Snapshot: MvccSnapshot[applied={T|T < 7299058620694953984 or (T in {7299058620694953984})}]
I20260620 23:05:33.574067 7368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:33.574205 7368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:33.574988 7368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8983e721f02400b8d50ff89d4c34167: Sending DeleteTablet for 3 replicas of tablet 68ee8c0c0cbb4644b2b1c6d34d706311
I20260620 23:05:33.575274 7628 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ee8c0c0cbb4644b2b1c6d34d706311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:33 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:52682
I20260620 23:05:33.575315 7494 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ee8c0c0cbb4644b2b1c6d34d706311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:33 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:50674
I20260620 23:05:33.575538 7869 tablet_replica.cc:330] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2: stopping tablet replica
I20260620 23:05:33.575642 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7431
I20260620 23:05:33.575646 7869 raft_consensus.cc:2229] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:33.575758 7869 raft_consensus.cc:2258] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 23b259fb919b421eb10d99e6603783c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:33.578886 7744 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ee8c0c0cbb4644b2b1c6d34d706311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:33 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:40190
I20260620 23:05:33.579106 7870 tablet_replica.cc:330] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: stopping tablet replica
I20260620 23:05:33.579128 7868 tablet_replica.cc:330] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: stopping tablet replica
I20260620 23:05:33.579231 7868 raft_consensus.cc:2229] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:33.579253 7870 raft_consensus.cc:2229] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:33.579344 7868 raft_consensus.cc:2258] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:33.579473 7870 raft_consensus.cc:2258] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:33.579784 7868 ts_tablet_manager.cc:1905] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:33.580026 7870 ts_tablet_manager.cc:1905] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:33.581713 7870 ts_tablet_manager.cc:1918] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:33.581754 7868 ts_tablet_manager.cc:1918] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:33.581826 7868 log.cc:1196] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/68ee8c0c0cbb4644b2b1c6d34d706311
I20260620 23:05:33.581933 7870 log.cc:1196] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/68ee8c0c0cbb4644b2b1c6d34d706311
I20260620 23:05:33.582137 7868 ts_tablet_manager.cc:1939] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 54cde0193bec4dd487bcdc7a34688d17: Deleting consensus metadata
I20260620 23:05:33.582365 7870 ts_tablet_manager.cc:1939] T 68ee8c0c0cbb4644b2b1c6d34d706311 P 3cdf2903aa9c435f817b2403c7bf1a54: Deleting consensus metadata
I20260620 23:05:33.582576 7353 catalog_manager.cc:4841] TS 54cde0193bec4dd487bcdc7a34688d17 (127.21.66.66:46273): tablet 68ee8c0c0cbb4644b2b1c6d34d706311 (table test-table [id=decaec500ba4446d9df1422a3b538b56]) successfully deleted
I20260620 23:05:33.582792 7353 catalog_manager.cc:4841] TS 3cdf2903aa9c435f817b2403c7bf1a54 (127.21.66.67:37557): tablet 68ee8c0c0cbb4644b2b1c6d34d706311 (table test-table [id=decaec500ba4446d9df1422a3b538b56]) successfully deleted
W20260620 23:05:33.583056 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:45225 (error 108)
I20260620 23:05:33.583578 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.583440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:45225 (local address 127.0.0.1:50686)
0620 23:05:33.583499 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.583543 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:45225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 23:05:33.583681 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7565
W20260620 23:05:33.585592 7355 catalog_manager.cc:4584] TS 23b259fb919b421eb10d99e6603783c2 (127.21.66.65:45225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68ee8c0c0cbb4644b2b1c6d34d706311: Network error: Client connection negotiation failed: client connection to 127.21.66.65:45225: connect: Connection refused (error 111)
I20260620 23:05:33.588970 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7699
I20260620 23:05:33.594223 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7337
2026-06-20T23:05:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2253 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:33Z Disabled control of system clock
I20260620 23:05:33.624140 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:43831
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:43831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:05:33.710456 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:33.710677 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:33.710698 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:33.711781 7875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:33.711817 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:33.711829 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:33.711841 7875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:33.711897 7875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:33.713258 7875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:43831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:33.713454 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:33.713682 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:33.718568 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.718628 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.718628 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:33.718773 7875 server_base.cc:1034] running on GCE node
I20260620 23:05:33.719043 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:33.719285 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:33.720428 7875 hybrid_clock.cc:648] HybridClock initialized: now 1781996733720406 us; error 33 us; skew 500 ppm
I20260620 23:05:33.724545 7875 webserver.cc:460] Webserver started at http://127.21.66.126:42735/ using document root <none> and password file <none>
I20260620 23:05:33.724771 7875 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:33.724814 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:33.724913 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:33.725967 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "8759a28a8725422fabf9b0d14f696506"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "a4a7fb433512198014ee21d146422d7a"
server_key_iv: "4885da737edaeeea7d647019ec205b76"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.726362 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "8759a28a8725422fabf9b0d14f696506"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "a4a7fb433512198014ee21d146422d7a"
server_key_iv: "4885da737edaeeea7d647019ec205b76"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.727695 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:33.728525 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.728751 7875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:33.728880 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "8759a28a8725422fabf9b0d14f696506"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "a4a7fb433512198014ee21d146422d7a"
server_key_iv: "4885da737edaeeea7d647019ec205b76"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.728963 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:33.738510 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:33.738888 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:33.739053 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:33.743563 7875 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:43831
I20260620 23:05:33.743593 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:43831 every 8 connection(s)
I20260620 23:05:33.743996 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:33.744971 7942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:33.747437 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506: Bootstrap starting.
I20260620 23:05:33.748104 7942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:33.748414 7942 log.cc:824] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:33.749248 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506: No bootstrap required, opened a new log
I20260620 23:05:33.751137 7942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8759a28a8725422fabf9b0d14f696506" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43831 } }
I20260620 23:05:33.751287 7942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:33.751312 7942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8759a28a8725422fabf9b0d14f696506, State: Initialized, Role: FOLLOWER
I20260620 23:05:33.751494 7942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8759a28a8725422fabf9b0d14f696506" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43831 } }
I20260620 23:05:33.751573 7942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:33.751608 7942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:33.751650 7942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:33.752202 7942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8759a28a8725422fabf9b0d14f696506" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43831 } }
I20260620 23:05:33.752326 7942 leader_election.cc:302] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8759a28a8725422fabf9b0d14f696506; no voters:
I20260620 23:05:33.752521 7942 leader_election.cc:288] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:33.752573 7945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:33.752805 7942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:33.752805 7945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [term 1 LEADER]: Becoming Leader. State: Replica: 8759a28a8725422fabf9b0d14f696506, State: Running, Role: LEADER
I20260620 23:05:33.752943 7945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8759a28a8725422fabf9b0d14f696506" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43831 } }
I20260620 23:05:33.753342 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7875
I20260620 23:05:33.753449 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:33.753439 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8759a28a8725422fabf9b0d14f696506. Latest consensus state: current_term: 1 leader_uuid: "8759a28a8725422fabf9b0d14f696506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8759a28a8725422fabf9b0d14f696506" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43831 } } }
I20260620 23:05:33.753518 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:33.753695 21769 external_mini_cluster.cc:1428] Setting key 8e8dd1691f3833aa3ec40bfb6c680750
I20260620 23:05:33.753786 7954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:33.753862 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8759a28a8725422fabf9b0d14f696506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8759a28a8725422fabf9b0d14f696506" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43831 } } }
I20260620 23:05:33.753917 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:33.754448 7954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:33.756456 7954 catalog_manager.cc:1347] Generated new cluster ID: bdb9b2e415b242b6a4f202f026ceff62
I20260620 23:05:33.756515 7954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:33.759487 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.754343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.21.66.126:43831)
0620 23:05:33.754529 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.754540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.754576 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:05:33.754718 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.754725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.754819 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.754982 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:33.755210 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.756500 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.757348 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.757804 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.757950 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.758745 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:33.758749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:33.758749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:33.758936 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:33.758981 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:33.758992 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:33.759068 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.759142 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.759325 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260620 23:05:33.787004 7954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:33.787585 7954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:33.804621 7954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506: Generated new TSK 0
I20260620 23:05:33.804888 7954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:33.809432 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:43831
--builtin_ntp_servers=127.21.66.84:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:05:33.899609 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:33.899853 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:33.899878 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:33.901176 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:33.901232 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:33.901262 7966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:05:33.902693 7966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:43831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:33.902940 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:33.903230 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:33.908802 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.908891 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:33.908834 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:33.909055 7966 server_base.cc:1034] running on GCE node
I20260620 23:05:33.909281 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:33.909533 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:33.910718 7966 hybrid_clock.cc:648] HybridClock initialized: now 1781996733910703 us; error 47 us; skew 500 ppm
I20260620 23:05:33.912160 7966 webserver.cc:460] Webserver started at http://127.21.66.65:34115/ using document root <none> and password file <none>
I20260620 23:05:33.912487 7966 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:33.912575 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:33.912740 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:33.913929 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "e72155220d5f48b89557936efe9dd464"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "3b6cda2b44e7e2e71dbe029b28040a5a"
server_key_iv: "72c30d60f28876ccd52487b196d5c60e"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.914361 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "e72155220d5f48b89557936efe9dd464"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "3b6cda2b44e7e2e71dbe029b28040a5a"
server_key_iv: "72c30d60f28876ccd52487b196d5c60e"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.915776 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:33.916692 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.916860 7966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:33.916945 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "e72155220d5f48b89557936efe9dd464"
format_stamp: "Formatted at 2026-06-20 23:05:33 on dist-test-slave-qgtk"
server_key: "3b6cda2b44e7e2e71dbe029b28040a5a"
server_key_iv: "72c30d60f28876ccd52487b196d5c60e"
server_key_version: "encryptionkey@0"
I20260620 23:05:33.917025 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:33.924134 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:33.924396 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:33.924494 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:33.924671 7966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:33.924995 7966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:33.925033 7966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.925055 7966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:33.925067 7966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:33.930720 7966 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:43099
I20260620 23:05:33.930853 8093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:43099 every 8 connection(s)
I20260620 23:05:33.931150 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:33.935415 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 7966
I20260620 23:05:33.935555 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:33.935783 21769 external_mini_cluster.cc:1428] Setting key 1146f0016ecdc8cd379428b1022e2070
I20260620 23:05:33.936770 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.931968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:33717 (local address 127.21.66.126:43831)
0620 23:05:33.932048 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:33.932050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:33.932295 (+ 245us) server_negotiation.cc:407] Connection header received
0620 23:05:33.932844 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:33.932846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:33.932862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:33.932930 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:33.933574 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.934100 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.935219 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.935451 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.935580 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.936507 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:33.936508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:33.936509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:33.936514 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:33.936529 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:33.936533 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:33.936582 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:33.936656 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 23:05:33.936692 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:05:33.936779 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:43831
--builtin_ntp_servers=127.21.66.84:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:05:33.937014 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:33.931953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:43831 (local address 127.21.66.65:33717)
0620 23:05:33.932170 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:33.932206 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:33.932705 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:33.932985 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:33.932988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:33.932996 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:33.933480 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.933484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.934268 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.934269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.935116 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:33.935118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:33.935588 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:33.935589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:33.935724 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:33.936342 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:33.936343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:33.936373 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:33.936399 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:33.936582 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:33.936587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:33.936590 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:05:33.936651 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 23:05:33.936761 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260620 23:05:33.937500 8094 heartbeater.cc:347] Connected to a master server at 127.21.66.126:43831
I20260620 23:05:33.937592 8094 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:33.937748 8094 heartbeater.cc:511] Master 127.21.66.126:43831 requested a full tablet report, sending...
I20260620 23:05:33.938380 7906 ts_manager.cc:194] Registered new tserver with Master: e72155220d5f48b89557936efe9dd464 (127.21.66.65:43099)
I20260620 23:05:33.939029 7906 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.65:33717
W20260620 23:05:34.020285 8097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:34.020507 8097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:34.020532 8097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:34.021925 8097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:34.021975 8097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:34.022006 8097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:05:34.023574 8097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:43831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:34.023851 8097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:34.024228 8097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:34.029892 8105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:34.029929 8102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:34.029918 8103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:34.030320 8097 server_base.cc:1034] running on GCE node
I20260620 23:05:34.030532 8097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:34.030766 8097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:34.031985 8097 hybrid_clock.cc:648] HybridClock initialized: now 1781996734031969 us; error 27 us; skew 500 ppm
I20260620 23:05:34.033455 8097 webserver.cc:460] Webserver started at http://127.21.66.66:38613/ using document root <none> and password file <none>
I20260620 23:05:34.033730 8097 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:34.033782 8097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:34.033921 8097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:34.035076 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "2b101648847a4318badc74c196a63744"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-qgtk"
server_key: "b1d4c396cde8589c09f0db92984d9d5c"
server_key_iv: "f4acb194c7beae8f602f16edcacdbdae"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.035449 8097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "2b101648847a4318badc74c196a63744"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-qgtk"
server_key: "b1d4c396cde8589c09f0db92984d9d5c"
server_key_iv: "f4acb194c7beae8f602f16edcacdbdae"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.036851 8097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:05:34.037750 8111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.037979 8097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:34.038053 8097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "2b101648847a4318badc74c196a63744"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-qgtk"
server_key: "b1d4c396cde8589c09f0db92984d9d5c"
server_key_iv: "f4acb194c7beae8f602f16edcacdbdae"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.038208 8097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:34.058315 8097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:34.058650 8097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:34.058811 8097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:34.059070 8097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:34.059422 8097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:34.059456 8097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.059478 8097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:34.059510 8097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.065935 8097 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:45333
I20260620 23:05:34.066033 8224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:45333 every 8 connection(s)
I20260620 23:05:34.066432 8097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:34.072130 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.067400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:44709 (local address 127.21.66.126:43831)
0620 23:05:34.067462 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.067464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.067688 (+ 224us) server_negotiation.cc:407] Connection header received
0620 23:05:34.068300 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.068302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.068315 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.068361 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.069089 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.069595 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.070657 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.070866 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.070946 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.071821 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.071822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.071823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.071828 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.071846 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.071850 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.071915 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.072017 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.072048 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:05:34.072376 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.067396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:43831 (local address 127.21.66.66:44709)
0620 23:05:34.067582 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.067617 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.068165 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.068442 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.068452 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.068461 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.068990 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.068994 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.069691 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.069692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.070567 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.070570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.070953 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.070954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.071092 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.071656 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.071657 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.071686 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.071709 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.071929 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.071935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.071937 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.072020 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.072124 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260620 23:05:34.072570 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 8097
I20260620 23:05:34.072750 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:34.072894 8225 heartbeater.cc:347] Connected to a master server at 127.21.66.126:43831
I20260620 23:05:34.072993 8225 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:34.073012 21769 external_mini_cluster.cc:1428] Setting key 9bfee9bce7c272b623daf1b8b267b776
I20260620 23:05:34.073184 8225 heartbeater.cc:511] Master 127.21.66.126:43831 requested a full tablet report, sending...
I20260620 23:05:34.073606 7905 ts_manager.cc:194] Registered new tserver with Master: 2b101648847a4318badc74c196a63744 (127.21.66.66:45333)
I20260620 23:05:34.073997 7905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.66:44709
I20260620 23:05:34.074059 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:43831
--builtin_ntp_servers=127.21.66.84:45737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:05:34.156677 8228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:34.156900 8228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:34.156924 8228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:34.158152 8228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:34.158196 8228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:34.158223 8228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:05:34.159586 8228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:43831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:34.159926 8228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:34.160249 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:34.165668 8236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:34.165694 8228 server_base.cc:1034] running on GCE node
W20260620 23:05:34.165724 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:34.165999 8233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:34.166249 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:34.166478 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:34.167640 8228 hybrid_clock.cc:648] HybridClock initialized: now 1781996734167626 us; error 25 us; skew 500 ppm
I20260620 23:05:34.169003 8228 webserver.cc:460] Webserver started at http://127.21.66.67:38425/ using document root <none> and password file <none>
I20260620 23:05:34.169214 8228 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:34.169261 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:34.169375 8228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:34.170457 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "1ea66a95174f4625a7b3c535d620f3c7"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-qgtk"
server_key: "9ae5f83ad4d4fa9b29cb198481e1d63c"
server_key_iv: "02348538be0699860e41703bb56bdbda"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.170760 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "1ea66a95174f4625a7b3c535d620f3c7"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-qgtk"
server_key: "9ae5f83ad4d4fa9b29cb198481e1d63c"
server_key_iv: "02348538be0699860e41703bb56bdbda"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.171999 8228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:34.172681 8242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.172931 8228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:34.173033 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "1ea66a95174f4625a7b3c535d620f3c7"
format_stamp: "Formatted at 2026-06-20 23:05:34 on dist-test-slave-qgtk"
server_key: "9ae5f83ad4d4fa9b29cb198481e1d63c"
server_key_iv: "02348538be0699860e41703bb56bdbda"
server_key_version: "encryptionkey@0"
I20260620 23:05:34.173111 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:34.185720 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:34.186079 8228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:34.186278 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:34.186525 8228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:34.186867 8228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:34.186929 8228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.186973 8228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:34.187004 8228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:34.193004 8228 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:36253
I20260620 23:05:34.193048 8355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:36253 every 8 connection(s)
I20260620 23:05:34.193394 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:34.198925 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.194184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:41921 (local address 127.21.66.126:43831)
0620 23:05:34.194246 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.194249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.194517 (+ 268us) server_negotiation.cc:407] Connection header received
0620 23:05:34.194991 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.194992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.195002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.195056 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.195734 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.196289 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.197404 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.197597 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.197659 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.198659 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.198661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.198661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.198666 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.198684 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.198689 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.198756 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.198826 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.198858 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:05:34.199095 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 8228
I20260620 23:05:34.199203 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:34.199307 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.194182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:43831 (local address 127.21.66.67:41921)
0620 23:05:34.194383 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.194420 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.194902 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.195111 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.195114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.195122 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:34.195622 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.195626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.196416 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.196417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.197299 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.197302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.197676 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.197677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.197792 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.198508 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:34.198508 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:34.198539 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:34.198564 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:34.198758 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:34.198760 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:34.198763 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.198812 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.198951 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260620 23:05:34.199486 21769 external_mini_cluster.cc:1428] Setting key b0cfd210fefed0b103e133aeabcbfc16
I20260620 23:05:34.199913 8356 heartbeater.cc:347] Connected to a master server at 127.21.66.126:43831
I20260620 23:05:34.200047 8356 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:34.200265 8356 heartbeater.cc:511] Master 127.21.66.126:43831 requested a full tablet report, sending...
I20260620 23:05:34.200680 7905 ts_manager.cc:194] Registered new tserver with Master: 1ea66a95174f4625a7b3c535d620f3c7 (127.21.66.67:36253)
I20260620 23:05:34.201092 7905 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.67:41921
I20260620 23:05:34.211547 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:34.216305 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.213010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.21.66.126:43831)
0620 23:05:34.213108 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.213110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.213201 (+ 91us) server_negotiation.cc:407] Connection header received
0620 23:05:34.213304 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.213305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.213319 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.213366 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:34.213550 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.214075 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.214947 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.215142 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.215218 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.215944 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:34.215945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:34.215945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:34.215951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:34.215965 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:34.215970 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:34.216084 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.216176 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.216187 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:05:34.218643 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:34.220305 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:34.226949 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.222725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.21.66.67:36253)
0620 23:05:34.223194 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.223199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.223210 (+ 11us) server_negotiation.cc:407] Connection header received
0620 23:05:34.223259 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.223261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.223276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.223511 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:34.223572 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.224357 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.225656 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.226524 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.226614 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.226726 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.226811 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.226850 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":87,"threads_started":1}
I20260620 23:05:34.226938 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.222587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:36253 (local address 127.0.0.1:53454)
0620 23:05:34.222980 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.222990 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.223033 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.223381 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.223382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.223385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:34.223435 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.223437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.224446 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.224448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.225560 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.225563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.226612 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.226613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.226696 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.226710 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.226793 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.226860 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":131,"threads_started":1}
I20260620 23:05:34.227103 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.222586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:45333 (local address 127.0.0.1:47650)
0620 23:05:34.222789 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.222929 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.223033 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.223290 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.223294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.223297 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:34.223377 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.223380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.225015 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.225016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.225998 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.226001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.226873 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.226874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.226932 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.226936 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.226966 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.227004 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260620 23:05:34.227461 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.222791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.21.66.66:45333)
0620 23:05:34.223064 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.223071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.223085 (+ 14us) server_negotiation.cc:407] Connection header received
0620 23:05:34.223136 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.223138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.223158 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.223472 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:34.223543 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.224919 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.226092 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.226774 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.227213 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.227304 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.227329 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.227367 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":96,"threads_started":1}
I20260620 23:05:34.227751 8290 tablet_service.cc:1467] Processing CreateTablet for tablet 320d2f1492e7460f9b53fe9a490615ae (DEFAULT_TABLE table=test-table [id=ac785d0bcb14411bb7c4b275c8d02240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:34.227996 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.222838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43099 (local address 127.0.0.1:52240)
0620 23:05:34.223371 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.223382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.223437 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.223644 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.223644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.223646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:34.223677 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.223678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.225052 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.225053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.226322 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.226326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.227771 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.227772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.227827 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.227831 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.227884 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.227924 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":79,"threads_started":1}
I20260620 23:05:34.228178 8290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d2f1492e7460f9b53fe9a490615ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:34.228204 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.223055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.21.66.65:43099)
0620 23:05:34.223236 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.223240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.223499 (+ 259us) server_negotiation.cc:407] Connection header received
0620 23:05:34.223545 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.223548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.223564 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.223647 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:34.223746 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.224959 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.226425 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.227663 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.227767 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.227849 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.227899 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.227934 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260620 23:05:34.230526 8159 tablet_service.cc:1467] Processing CreateTablet for tablet 320d2f1492e7460f9b53fe9a490615ae (DEFAULT_TABLE table=test-table [id=ac785d0bcb14411bb7c4b275c8d02240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:34.230888 8159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d2f1492e7460f9b53fe9a490615ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:34.231375 8028 tablet_service.cc:1467] Processing CreateTablet for tablet 320d2f1492e7460f9b53fe9a490615ae (DEFAULT_TABLE table=test-table [id=ac785d0bcb14411bb7c4b275c8d02240]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:34.231719 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 320d2f1492e7460f9b53fe9a490615ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:34.234411 8371 tablet_bootstrap.cc:492] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Bootstrap starting.
I20260620 23:05:34.235049 8372 tablet_bootstrap.cc:492] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464: Bootstrap starting.
I20260620 23:05:34.235169 8370 tablet_bootstrap.cc:492] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Bootstrap starting.
I20260620 23:05:34.235250 8371 tablet_bootstrap.cc:654] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:34.235559 8371 log.cc:824] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:34.235714 8372 tablet_bootstrap.cc:654] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:34.235754 8370 tablet_bootstrap.cc:654] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:34.236053 8372 log.cc:824] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:34.236053 8370 log.cc:824] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:34.236479 8371 tablet_bootstrap.cc:492] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: No bootstrap required, opened a new log
I20260620 23:05:34.236563 8371 ts_tablet_manager.cc:1397] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:34.236770 8370 tablet_bootstrap.cc:492] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: No bootstrap required, opened a new log
I20260620 23:05:34.236852 8370 ts_tablet_manager.cc:1397] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:34.236871 8372 tablet_bootstrap.cc:492] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464: No bootstrap required, opened a new log
I20260620 23:05:34.236941 8372 ts_tablet_manager.cc:1397] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:34.238863 8370 raft_consensus.cc:348] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.239035 8370 raft_consensus.cc:374] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:34.239075 8370 raft_consensus.cc:729] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea66a95174f4625a7b3c535d620f3c7, State: Initialized, Role: FOLLOWER
I20260620 23:05:34.239223 8370 consensus_queue.cc:260] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.239491 8370 ts_tablet_manager.cc:1428] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:34.239619 8356 heartbeater.cc:503] Master 127.21.66.126:43831 was elected leader, sending a full tablet report...
I20260620 23:05:34.239585 8371 raft_consensus.cc:348] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.239759 8371 raft_consensus.cc:374] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:34.239817 8371 raft_consensus.cc:729] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b101648847a4318badc74c196a63744, State: Initialized, Role: FOLLOWER
I20260620 23:05:34.239948 8371 consensus_queue.cc:260] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.239899 8372 raft_consensus.cc:348] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.240242 8371 ts_tablet_manager.cc:1428] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 23:05:34.240257 8372 raft_consensus.cc:374] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:34.240302 8372 raft_consensus.cc:729] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72155220d5f48b89557936efe9dd464, State: Initialized, Role: FOLLOWER
I20260620 23:05:34.240509 8225 heartbeater.cc:503] Master 127.21.66.126:43831 was elected leader, sending a full tablet report...
I20260620 23:05:34.240763 8372 consensus_queue.cc:260] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.241128 8372 ts_tablet_manager.cc:1428] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 23:05:34.241264 8094 heartbeater.cc:503] Master 127.21.66.126:43831 was elected leader, sending a full tablet report...
I20260620 23:05:34.262585 8376 raft_consensus.cc:482] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:34.262797 8376 raft_consensus.cc:504] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.263186 8376 leader_election.cc:288] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b101648847a4318badc74c196a63744 (127.21.66.66:45333), e72155220d5f48b89557936efe9dd464 (127.21.66.65:43099)
I20260620 23:05:34.266434 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.263412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:45333 (local address 127.21.66.67:37757)
0620 23:05:34.263485 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.263497 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.263545 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.263759 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.263760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.263763 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:34.263848 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.263850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.264564 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.264565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.265359 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.265361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.266100 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.266101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.266197 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.266260 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.266320 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.266363 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1}
I20260620 23:05:34.266513 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.263459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:37757 (local address 127.21.66.66:45333)
0620 23:05:34.263522 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.263524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.263536 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:05:34.263627 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.263628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.263642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.263714 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:34.263926 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.264465 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.265474 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.265926 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.265983 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.265989 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.266360 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.266409 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:05:34.266706 8369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.263515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:39689 (local address 127.21.66.65:43099)
0620 23:05:34.263560 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.263562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.263741 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:05:34.263775 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.263776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.263789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.263864 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:34.264029 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.264474 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.265660 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.266251 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.266331 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.266338 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.266575 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.266632 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 23:05:34.266920 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d2f1492e7460f9b53fe9a490615ae" candidate_uuid: "1ea66a95174f4625a7b3c535d620f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b101648847a4318badc74c196a63744" is_pre_election: true
I20260620 23:05:34.267030 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.263416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43099 (local address 127.21.66.67:39689)
0620 23:05:34.263660 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.263671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:34.263729 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:34.263868 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:34.263869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:34.263870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:34.263959 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.263961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.264577 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.264578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.265549 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:34.265552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.266363 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.266364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:34.266451 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.266455 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:34.266880 (+ 425us) client_negotiation.cc:240] Negotiation successful
0620 23:05:34.266928 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":82,"thread_start_us":103,"threads_started":1}
I20260620 23:05:34.267112 8179 raft_consensus.cc:2454] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea66a95174f4625a7b3c535d620f3c7 in term 0.
I20260620 23:05:34.267349 8246 leader_election.cc:302] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea66a95174f4625a7b3c535d620f3c7, 2b101648847a4318badc74c196a63744; no voters:
I20260620 23:05:34.267413 8048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d2f1492e7460f9b53fe9a490615ae" candidate_uuid: "1ea66a95174f4625a7b3c535d620f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72155220d5f48b89557936efe9dd464" is_pre_election: true
I20260620 23:05:34.267477 8376 raft_consensus.cc:2781] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:34.267524 8376 raft_consensus.cc:482] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:34.267571 8376 raft_consensus.cc:3037] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:34.267590 8048 raft_consensus.cc:2454] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea66a95174f4625a7b3c535d620f3c7 in term 0.
I20260620 23:05:34.268308 8376 raft_consensus.cc:504] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.268627 8048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d2f1492e7460f9b53fe9a490615ae" candidate_uuid: "1ea66a95174f4625a7b3c535d620f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e72155220d5f48b89557936efe9dd464"
I20260620 23:05:34.268628 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "320d2f1492e7460f9b53fe9a490615ae" candidate_uuid: "1ea66a95174f4625a7b3c535d620f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b101648847a4318badc74c196a63744"
I20260620 23:05:34.268711 8048 raft_consensus.cc:3037] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:34.268718 8179 raft_consensus.cc:3037] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:34.269474 8048 raft_consensus.cc:2454] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea66a95174f4625a7b3c535d620f3c7 in term 1.
I20260620 23:05:34.269570 8376 leader_election.cc:288] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [CANDIDATE]: Term 1 election: Requested vote from peers 2b101648847a4318badc74c196a63744 (127.21.66.66:45333), e72155220d5f48b89557936efe9dd464 (127.21.66.65:43099)
I20260620 23:05:34.269639 8179 raft_consensus.cc:2454] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea66a95174f4625a7b3c535d620f3c7 in term 1.
I20260620 23:05:34.269672 8245 leader_election.cc:302] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea66a95174f4625a7b3c535d620f3c7, e72155220d5f48b89557936efe9dd464; no voters:
I20260620 23:05:34.269796 8376 raft_consensus.cc:2781] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:34.269946 8376 raft_consensus.cc:686] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea66a95174f4625a7b3c535d620f3c7, State: Running, Role: LEADER
I20260620 23:05:34.270040 8376 consensus_queue.cc:237] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } }
I20260620 23:05:34.271052 7906 catalog_manager.cc:5495] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ea66a95174f4625a7b3c535d620f3c7 (127.21.66.67). New cstate: current_term: 1 leader_uuid: "1ea66a95174f4625a7b3c535d620f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea66a95174f4625a7b3c535d620f3c7" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:34.287376 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.284701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.21.66.67:36253)
0620 23:05:34.284762 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:34.284764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:34.284779 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:05:34.284902 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:34.284903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:34.284930 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:34.285014 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:34.285135 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.285637 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.286573 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:34.286786 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:34.286868 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:34.286992 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:34.287095 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:34.287195 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:34.287267 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 23:05:34.287280 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 23:05:34.289713 8048 raft_consensus.cc:1261] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 1 FOLLOWER]: Refusing update from remote peer 1ea66a95174f4625a7b3c535d620f3c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:34.289786 8179 raft_consensus.cc:1261] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 1 FOLLOWER]: Refusing update from remote peer 1ea66a95174f4625a7b3c535d620f3c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:34.290195 8380 consensus_queue.cc:1035] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b101648847a4318badc74c196a63744" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:34.290346 8376 consensus_queue.cc:1035] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e72155220d5f48b89557936efe9dd464" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:34.295761 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:34.295883 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:34.296694 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8759a28a8725422fabf9b0d14f696506: Sending DeleteTablet for 3 replicas of tablet 320d2f1492e7460f9b53fe9a490615ae
I20260620 23:05:34.297176 8028 tablet_service.cc:1514] Processing DeleteTablet for tablet 320d2f1492e7460f9b53fe9a490615ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:52240
I20260620 23:05:34.297186 8159 tablet_service.cc:1514] Processing DeleteTablet for tablet 320d2f1492e7460f9b53fe9a490615ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:47650
I20260620 23:05:34.297209 8290 tablet_service.cc:1514] Processing DeleteTablet for tablet 320d2f1492e7460f9b53fe9a490615ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:34 UTC) from {username='slave'} at 127.0.0.1:53454
I20260620 23:05:34.297529 8394 tablet_replica.cc:330] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464: stopping tablet replica
I20260620 23:05:34.297686 8394 raft_consensus.cc:2229] T 320d2f1492e7460f9b53fe9a490615ae P e72155220d5f48b89557936efe9dd464 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:34.297753 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7966
I20260620 23:05:34.298660 8393 tablet_replica.cc:330] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: stopping tablet replica
I20260620 23:05:34.298802 8393 raft_consensus.cc:2229] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:34.298929 8393 raft_consensus.cc:2258] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:34.299216 8395 tablet_replica.cc:330] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: stopping tablet replica
I20260620 23:05:34.299319 8395 raft_consensus.cc:2229] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:34.299448 8395 raft_consensus.cc:2258] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:34.299834 8393 ts_tablet_manager.cc:1905] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:34.301465 8393 ts_tablet_manager.cc:1918] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:34.301570 8393 log.cc:1196] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/320d2f1492e7460f9b53fe9a490615ae
I20260620 23:05:34.301863 8393 ts_tablet_manager.cc:1939] T 320d2f1492e7460f9b53fe9a490615ae P 2b101648847a4318badc74c196a63744: Deleting consensus metadata
I20260620 23:05:34.302464 7893 catalog_manager.cc:4841] TS 2b101648847a4318badc74c196a63744 (127.21.66.66:45333): tablet 320d2f1492e7460f9b53fe9a490615ae (table test-table [id=ac785d0bcb14411bb7c4b275c8d02240]) successfully deleted
I20260620 23:05:34.302619 8395 ts_tablet_manager.cc:1905] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:34.304071 8395 ts_tablet_manager.cc:1918] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:34.304163 8395 log.cc:1196] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/320d2f1492e7460f9b53fe9a490615ae
I20260620 23:05:34.304445 8395 ts_tablet_manager.cc:1939] T 320d2f1492e7460f9b53fe9a490615ae P 1ea66a95174f4625a7b3c535d620f3c7: Deleting consensus metadata
I20260620 23:05:34.304875 7892 catalog_manager.cc:4841] TS 1ea66a95174f4625a7b3c535d620f3c7 (127.21.66.67:36253): tablet 320d2f1492e7460f9b53fe9a490615ae (table test-table [id=ac785d0bcb14411bb7c4b275c8d02240]) successfully deleted
W20260620 23:05:34.305866 7892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:43099 (error 108)
I20260620 23:05:34.306267 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 8097
I20260620 23:05:34.307502 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:34.307312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:43099 (local address 127.0.0.1:52252)
0620 23:05:34.307370 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:34.307418 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 23:05:34.307729 7892 catalog_manager.cc:4584] TS e72155220d5f48b89557936efe9dd464 (127.21.66.65:43099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 320d2f1492e7460f9b53fe9a490615ae: Network error: Client connection negotiation failed: client connection to 127.21.66.65:43099: connect: Connection refused (error 111)
I20260620 23:05:34.311595 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 8228
I20260620 23:05:34.317158 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 7875
2026-06-20T23:05:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (720 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:34 dist-test-slave-qgtk krb5kdc[8399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:34 dist-test-slave-qgtk krb5kdc[8399](info): set up 2 sockets
Jun 20 23:05:34 dist-test-slave-qgtk krb5kdc[8399](info): commencing operation
krb5kdc: starting...
W20260620 23:05:35.398898 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:35 dist-test-slave-qgtk krb5kdc[8399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:05:35.528977 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44103
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:44103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:35.611655 8412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:35.611918 8412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:35.611959 8412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:35.613137 8412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:35.613207 8412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:35.613230 8412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:35.613252 8412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:35.613281 8412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:35.614825 8412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:44103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:35.615113 8412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:35.615347 8412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:35.620563 8420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:35.620589 8417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:35.620584 8418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:35.621282 8412 server_base.cc:1034] running on GCE node
I20260620 23:05:35.621570 8412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:35.621965 8412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:35.623142 8412 hybrid_clock.cc:648] HybridClock initialized: now 1781996735623122 us; error 40 us; skew 500 ppm
Jun 20 23:05:35 dist-test-slave-qgtk krb5kdc[8399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996735, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:35.626012 8412 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:35.626513 8412 webserver.cc:460] Webserver started at http://127.21.66.126:45791/ using document root <none> and password file <none>
I20260620 23:05:35.626745 8412 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:35.626792 8412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:35.626899 8412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:35.628031 8412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "fc23ca4f93f344d29ccbd478a311676d"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "d32eff80edde5a511d7bd4e4467374c7"
server_key_iv: "02d5348448572f4342d985b37caa188e"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.628369 8412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "fc23ca4f93f344d29ccbd478a311676d"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "d32eff80edde5a511d7bd4e4467374c7"
server_key_iv: "02d5348448572f4342d985b37caa188e"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.629737 8412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:05:35.630544 8427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:35.630713 8412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:35.630821 8412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "fc23ca4f93f344d29ccbd478a311676d"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "d32eff80edde5a511d7bd4e4467374c7"
server_key_iv: "02d5348448572f4342d985b37caa188e"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.630899 8412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:35.642925 8412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:35.645171 8412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:35.645345 8412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:35.649593 8412 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:44103
I20260620 23:05:35.649628 8479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:44103 every 8 connection(s)
I20260620 23:05:35.650002 8412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:35.651022 8480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:35.653318 8480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d: Bootstrap starting.
I20260620 23:05:35.654081 8480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:35.654358 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 8412
I20260620 23:05:35.654502 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:35.654474 8480 log.cc:824] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:35.654750 21769 external_mini_cluster.cc:1428] Setting key f904d5aac7f4707b3751fece6c595eed
I20260620 23:05:35.655365 8480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d: No bootstrap required, opened a new log
I20260620 23:05:35.657066 8480 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23ca4f93f344d29ccbd478a311676d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44103 } }
I20260620 23:05:35.657212 8480 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:35.657235 8480 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc23ca4f93f344d29ccbd478a311676d, State: Initialized, Role: FOLLOWER
I20260620 23:05:35.657366 8480 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23ca4f93f344d29ccbd478a311676d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44103 } }
I20260620 23:05:35.657462 8480 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:35.657507 8480 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:35.657557 8480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:35.658545 8480 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23ca4f93f344d29ccbd478a311676d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44103 } }
I20260620 23:05:35.658670 8480 leader_election.cc:302] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc23ca4f93f344d29ccbd478a311676d; no voters:
I20260620 23:05:35.658850 8480 leader_election.cc:288] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:35.658912 8485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:35.659071 8485 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [term 1 LEADER]: Becoming Leader. State: Replica: fc23ca4f93f344d29ccbd478a311676d, State: Running, Role: LEADER
I20260620 23:05:35.659122 8480 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:35.659168 8485 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23ca4f93f344d29ccbd478a311676d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44103 } }
I20260620 23:05:35.659504 8485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc23ca4f93f344d29ccbd478a311676d. Latest consensus state: current_term: 1 leader_uuid: "fc23ca4f93f344d29ccbd478a311676d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23ca4f93f344d29ccbd478a311676d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44103 } } }
I20260620 23:05:35.659572 8485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:35.659523 8486 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc23ca4f93f344d29ccbd478a311676d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc23ca4f93f344d29ccbd478a311676d" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44103 } } }
I20260620 23:05:35.659785 8491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:35.660377 8491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:35.659768 8486 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:35.663072 8491 catalog_manager.cc:1347] Generated new cluster ID: 61ce0a4e11464e16b9b6b51078fba00c
I20260620 23:05:35.663134 8491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:05:35 dist-test-slave-qgtk krb5kdc[8399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:35.668249 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:35.655320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.21.66.126:44103)
0620 23:05:35.655541 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:35.655551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:35.655585 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:05:35.657084 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:35.657094 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:35.657159 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:35.657362 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:35.658272 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.659787 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.661072 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.661742 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.661864 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:35.664515 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:35.664523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:35.664523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:35.664534 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:35.666361 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:35.666744 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:35.666747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:35.666747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:35.666854 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:35.667070 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:35.667071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:35.667071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:35.667314 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:35.667898 (+ 584us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:35.667964 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:05:35.668081 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260620 23:05:35.689239 8491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:35.694465 8491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:35.699786 8491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d: Generated new TSK 0
I20260620 23:05:35.700021 8491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.66.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
Entry for principal kudu/127.21.66.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab.
I20260620 23:05:35.732664 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44103
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:35.817703 8506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:35.817909 8506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:35.817934 8506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:35.819211 8506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:35.819258 8506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:35.819285 8506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:05:35.820863 8506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.65.keytab
--principal=kudu/127.21.66.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:35.821117 8506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:35.821382 8506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:35.826568 8514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:35.826614 8512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:35.826737 8506 server_base.cc:1034] running on GCE node
W20260620 23:05:35.826767 8511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:35.827147 8506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:35.827471 8506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:35.828688 8506 hybrid_clock.cc:648] HybridClock initialized: now 1781996735828669 us; error 30 us; skew 500 ppm
Jun 20 23:05:35 dist-test-slave-qgtk krb5kdc[8399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996735, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:35.831559 8506 init.cc:377] Logged in from keytab as kudu/127.21.66.65@KRBTEST.COM (short username kudu)
I20260620 23:05:35.832149 8506 webserver.cc:460] Webserver started at http://127.21.66.65:40493/ using document root <none> and password file <none>
I20260620 23:05:35.832404 8506 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:35.832475 8506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:35.832623 8506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:35.833799 8506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "f96ac9253278449b8741d53f57865f8b"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "989277d3b0ea70cb1d07c5ab8e0eca30"
server_key_iv: "f4deb84661bcc78c64f89870b314d1d2"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.834225 8506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "f96ac9253278449b8741d53f57865f8b"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "989277d3b0ea70cb1d07c5ab8e0eca30"
server_key_iv: "f4deb84661bcc78c64f89870b314d1d2"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.835840 8506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:35.836720 8521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:35.836979 8506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:35.837086 8506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "f96ac9253278449b8741d53f57865f8b"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "989277d3b0ea70cb1d07c5ab8e0eca30"
server_key_iv: "f4deb84661bcc78c64f89870b314d1d2"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.837162 8506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:35.854017 8506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:35.856045 8506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:35.856213 8506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:35.856456 8506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:35.856761 8506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:35.856817 8506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:35.856887 8506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:35.856920 8506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:35.863459 8506 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:42439
I20260620 23:05:35.863508 8634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:42439 every 8 connection(s)
I20260620 23:05:35.863897 8506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:35.868117 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 8506
I20260620 23:05:35.868258 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:35.868496 21769 external_mini_cluster.cc:1428] Setting key b2b85df99ac05ae1372def81a424e01a
Jun 20 23:05:35 dist-test-slave-qgtk krb5kdc[8399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996735, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.65@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:35.874801 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:35.864787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:34449 (local address 127.21.66.126:44103)
0620 23:05:35.864848 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:35.864850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:35.865051 (+ 201us) server_negotiation.cc:407] Connection header received
0620 23:05:35.865606 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:35.865608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:35.865623 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:35.865690 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:35.866788 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.867294 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.868329 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.868549 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.868636 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:35.871374 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:35.871376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:35.871377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:35.871385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:35.873117 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:35.873659 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:35.873660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:35.873660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:35.873716 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:35.874050 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:35.874051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:35.874051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:35.874228 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:35.874290 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:35.874658 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:05:35.874733 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 23:05:35.875087 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:35.864709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44103 (local address 127.21.66.65:34449)
0620 23:05:35.864902 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:35.864949 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:35.865471 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:35.865800 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:35.865804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:35.866488 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:35.866689 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:35.866695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.867401 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.867401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:35.868196 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:35.868197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:35.868661 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:35.868663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:35.868784 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:35.869412 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:35.869552 (+ 140us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:35.871266 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:35.873199 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:35.873206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:35.873209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:35.873536 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:35.873803 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:35.873804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:35.873805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:35.873969 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:35.874296 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:35.874303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:35.874497 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 23:05:35.874628 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 23:05:35.874833 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260620 23:05:35.875628 8635 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44103
I20260620 23:05:35.875738 8635 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:35.875972 8635 heartbeater.cc:511] Master 127.21.66.126:44103 requested a full tablet report, sending...
I20260620 23:05:35.876564 8444 ts_manager.cc:194] Registered new tserver with Master: f96ac9253278449b8741d53f57865f8b (127.21.66.65:42439)
I20260620 23:05:35.877182 8444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.65@KRBTEST.COM'} at 127.21.66.65:34449
WARNING: no policy specified for kudu/127.21.66.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
Entry for principal kudu/127.21.66.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab.
I20260620 23:05:35.897115 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44103
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:35.979463 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:35.979724 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:35.979768 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:35.980957 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:35.981029 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:35.981076 8640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:05:35.982690 8640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.66.keytab
--principal=kudu/127.21.66.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:35.982977 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:35.983219 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:35.988253 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:35.988277 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:35.988253 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:35.988973 8640 server_base.cc:1034] running on GCE node
I20260620 23:05:35.989220 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:35.989509 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:35.990669 8640 hybrid_clock.cc:648] HybridClock initialized: now 1781996735990653 us; error 32 us; skew 500 ppm
Jun 20 23:05:35 dist-test-slave-qgtk krb5kdc[8399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996735, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:35.993424 8640 init.cc:377] Logged in from keytab as kudu/127.21.66.66@KRBTEST.COM (short username kudu)
I20260620 23:05:35.993973 8640 webserver.cc:460] Webserver started at http://127.21.66.66:42643/ using document root <none> and password file <none>
I20260620 23:05:35.994268 8640 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:35.994344 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:35.994482 8640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:35.995548 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "87d9dd01f9b84b559ee4cd2dc76086a2"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "e90cf4cae342b5acc1f913c8203a1c51"
server_key_iv: "0fa6e7bab05ce07d75fedc0b6672b171"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.995914 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "87d9dd01f9b84b559ee4cd2dc76086a2"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "e90cf4cae342b5acc1f913c8203a1c51"
server_key_iv: "0fa6e7bab05ce07d75fedc0b6672b171"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.997272 8640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:35.998255 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:35.998485 8640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:35.998595 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "87d9dd01f9b84b559ee4cd2dc76086a2"
format_stamp: "Formatted at 2026-06-20 23:05:35 on dist-test-slave-qgtk"
server_key: "e90cf4cae342b5acc1f913c8203a1c51"
server_key_iv: "0fa6e7bab05ce07d75fedc0b6672b171"
server_key_version: "encryptionkey@0"
I20260620 23:05:35.998692 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.017966 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.020344 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.020536 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.020792 8640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:36.021121 8640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:36.021175 8640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.021219 8640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:36.021260 8640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.027554 8640 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:42135
I20260620 23:05:36.027598 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:42135 every 8 connection(s)
I20260620 23:05:36.027926 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:36.032493 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 8640
I20260620 23:05:36.032630 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:36.032832 21769 external_mini_cluster.cc:1428] Setting key c326dee0c9689f86ebd339e20a10367b
Jun 20 23:05:36 dist-test-slave-qgtk krb5kdc[8399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996735, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.66@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:36.038504 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.028629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:60341 (local address 127.21.66.126:44103)
0620 23:05:36.028686 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.028688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.028886 (+ 198us) server_negotiation.cc:407] Connection header received
0620 23:05:36.029390 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.029392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.029404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.029473 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.030640 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.031140 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.032281 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.032601 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.032679 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.035352 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.035353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.035354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:36.035363 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.036883 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:36.037393 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:36.037394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:36.037394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:36.037449 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:36.037771 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:36.037772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:36.037772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:36.037894 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.037961 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.038359 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.038436 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 23:05:36.038810 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.028589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44103 (local address 127.21.66.66:60341)
0620 23:05:36.028755 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.028797 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.029291 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.029550 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.029553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.030347 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:36.030555 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.030560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.031235 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.031236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.032164 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.032166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.032727 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.032728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.032843 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.033471 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:36.033482 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:36.035239 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:36.036954 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:36.036961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:36.036964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:36.037300 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:36.037520 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:36.037521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:36.037521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:36.037666 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:36.037971 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:36.037978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:36.038183 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.038310 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.038509 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260620 23:05:36.039530 8769 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44103
I20260620 23:05:36.039640 8769 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:36.039860 8769 heartbeater.cc:511] Master 127.21.66.126:44103 requested a full tablet report, sending...
I20260620 23:05:36.040251 8444 ts_manager.cc:194] Registered new tserver with Master: 87d9dd01f9b84b559ee4cd2dc76086a2 (127.21.66.66:42135)
I20260620 23:05:36.040663 8444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.66@KRBTEST.COM'} at 127.21.66.66:60341
WARNING: no policy specified for kudu/127.21.66.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
Entry for principal kudu/127.21.66.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab.
I20260620 23:05:36.059401 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44103
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:36.143417 8774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:36.143653 8774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:36.143678 8774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:36.144886 8774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:36.144934 8774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:36.144961 8774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:05:36.146554 8774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.67.keytab
--principal=kudu/127.21.66.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:36.146826 8774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:36.147089 8774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:36.152256 8780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.152309 8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.152320 8779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.152475 8774 server_base.cc:1034] running on GCE node
I20260620 23:05:36.152697 8774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:36.152956 8774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:36.154214 8774 hybrid_clock.cc:648] HybridClock initialized: now 1781996736154194 us; error 36 us; skew 500 ppm
Jun 20 23:05:36 dist-test-slave-qgtk krb5kdc[8399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996736, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:36.157023 8774 init.cc:377] Logged in from keytab as kudu/127.21.66.67@KRBTEST.COM (short username kudu)
I20260620 23:05:36.157519 8774 webserver.cc:460] Webserver started at http://127.21.66.67:39971/ using document root <none> and password file <none>
I20260620 23:05:36.157786 8774 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:36.157861 8774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:36.158023 8774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:36.159274 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "5b80ce9b38a54a8e8d282b50a0e7850f"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "58e8672e4304999f4cb4a965cbc0a725"
server_key_iv: "8867f76201bea0bf44c741590490533e"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.159735 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "5b80ce9b38a54a8e8d282b50a0e7850f"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "58e8672e4304999f4cb4a965cbc0a725"
server_key_iv: "8867f76201bea0bf44c741590490533e"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.161446 8774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:36.162395 8789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.162616 8774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:36.162731 8774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "5b80ce9b38a54a8e8d282b50a0e7850f"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "58e8672e4304999f4cb4a965cbc0a725"
server_key_iv: "8867f76201bea0bf44c741590490533e"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.162889 8774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.174434 8774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.177102 8774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.177291 8774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.177560 8774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:36.177920 8774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:36.177982 8774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.178032 8774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:36.178061 8774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.184710 8774 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:44147
I20260620 23:05:36.184742 8902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:44147 every 8 connection(s)
I20260620 23:05:36.185185 8774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
Jun 20 23:05:36 dist-test-slave-qgtk krb5kdc[8399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996736, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.67@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:36.195195 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 8774
I20260620 23:05:36.195338 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:36.195523 21769 external_mini_cluster.cc:1428] Setting key 72c24d04692eb3b5669e834fe1ea8d0f
I20260620 23:05:36.196699 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.186045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:40805 (local address 127.21.66.126:44103)
0620 23:05:36.186192 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.186195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.186412 (+ 217us) server_negotiation.cc:407] Connection header received
0620 23:05:36.186902 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.186903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.186915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.186980 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.187969 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.188502 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.189511 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.189887 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.189957 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.192697 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.192699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.192699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:36.192706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.194525 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:36.194962 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:36.194963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:36.194963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:36.195018 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:36.195783 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:36.195785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:36.195785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:36.195962 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.196041 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.196466 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.196575 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 23:05:36.196889 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.186018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44103 (local address 127.21.66.67:40805)
0620 23:05:36.186246 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.186284 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.186783 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.187060 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.187063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.187654 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:36.187860 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.187866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.188589 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.188590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.189420 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.189423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.189988 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.189989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.190195 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.190878 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:05:36.190887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:36.192551 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:36.194621 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:36.194629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:36.194632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:36.194887 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:36.195487 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:05:36.195488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:05:36.195488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:05:36.195674 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:05:36.196051 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:36.196060 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:36.196279 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.196412 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.196614 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":126,"threads_started":1}
I20260620 23:05:36.197423 8903 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44103
I20260620 23:05:36.197532 8903 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:36.197718 8903 heartbeater.cc:511] Master 127.21.66.126:44103 requested a full tablet report, sending...
I20260620 23:05:36.198189 8444 ts_manager.cc:194] Registered new tserver with Master: 5b80ce9b38a54a8e8d282b50a0e7850f (127.21.66.67:44147)
I20260620 23:05:36.198597 8444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.66.67@KRBTEST.COM'} at 127.21.66.67:40805
I20260620 23:05:36.207831 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:36.216213 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.209268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.21.66.126:44103)
0620 23:05:36.209367 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.209370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.209477 (+ 107us) server_negotiation.cc:407] Connection header received
0620 23:05:36.209570 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.209571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.209582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.209660 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.210395 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.210940 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.211789 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.212008 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.212086 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.213486 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.213488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.213488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:36.213497 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.214967 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:36.215363 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:36.215364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:36.215365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:36.215423 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:36.215623 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:36.215624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:36.215624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:36.215793 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.215903 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.216012 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.216092 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:05:36.218194 8444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:36.219951 8444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:36.227998 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.222649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:44147 (local address 127.0.0.1:37522)
0620 23:05:36.223242 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.223249 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.223341 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.223667 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.223668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.223669 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.223923 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.223927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.225469 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.225470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.226678 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.226681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.227678 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.227679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.227760 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.227774 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.227836 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.227900 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":223,"thread_start_us":109,"threads_started":1}
I20260620 23:05:36.228215 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.222645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42439 (local address 127.0.0.1:45400)
0620 23:05:36.223016 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.223060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.223138 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.223660 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.223664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.223667 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.223906 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.223909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.225445 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.225446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.226951 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.226954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.227819 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.227819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.227884 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.227888 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.227930 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.227965 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":131,"thread_start_us":66,"threads_started":1}
I20260620 23:05:36.228300 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.223087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.21.66.67:44147)
0620 23:05:36.223287 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.223295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.223367 (+ 72us) server_negotiation.cc:407] Connection header received
0620 23:05:36.223449 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.223452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.223467 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.223555 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.224026 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.225329 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.226781 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.227567 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.227670 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.228135 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.228166 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.228212 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260620 23:05:36.228402 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.222976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.21.66.66:42135)
0620 23:05:36.223212 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.223217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.223956 (+ 739us) server_negotiation.cc:407] Connection header received
0620 23:05:36.224017 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.224020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.224036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.224120 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.224236 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.225367 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.227043 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.227843 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.227910 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.227970 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.228261 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.228310 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260620 23:05:36.228624 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.222649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:42135 (local address 127.0.0.1:48782)
0620 23:05:36.223860 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.223873 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.223940 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.224132 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.224134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.224136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.224171 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.224172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.225636 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.225637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.226951 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.226954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.228006 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.228007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.228085 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.228090 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.228144 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.228212 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":39,"thread_start_us":114,"threads_started":1}
I20260620 23:05:36.228691 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.222755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.21.66.65:42439)
0620 23:05:36.223001 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.223006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.223358 (+ 352us) server_negotiation.cc:407] Connection header received
0620 23:05:36.223435 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.223439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.223457 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.223555 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.224017 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.225276 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.227053 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.227624 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.227713 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.227786 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.228038 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.228616 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":140,"threads_started":1}
I20260620 23:05:36.229064 8837 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab8314362c4964b2d14f82d46dd6a2 (DEFAULT_TABLE table=test-table [id=6a2a9107e7764962bf42241e29d8a29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:36.229271 8569 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab8314362c4964b2d14f82d46dd6a2 (DEFAULT_TABLE table=test-table [id=6a2a9107e7764962bf42241e29d8a29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:36.229455 8837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab8314362c4964b2d14f82d46dd6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.229342 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab8314362c4964b2d14f82d46dd6a2 (DEFAULT_TABLE table=test-table [id=6a2a9107e7764962bf42241e29d8a29d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:36.229557 8569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab8314362c4964b2d14f82d46dd6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.229746 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab8314362c4964b2d14f82d46dd6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.232615 8920 tablet_bootstrap.cc:492] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Bootstrap starting.
I20260620 23:05:36.233229 8922 tablet_bootstrap.cc:492] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b: Bootstrap starting.
I20260620 23:05:36.233335 8921 tablet_bootstrap.cc:492] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Bootstrap starting.
I20260620 23:05:36.233666 8920 tablet_bootstrap.cc:654] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.233942 8922 tablet_bootstrap.cc:654] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.234182 8921 tablet_bootstrap.cc:654] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.234457 8920 log.cc:824] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.234527 8922 log.cc:824] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.234853 8921 log.cc:824] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.235373 8922 tablet_bootstrap.cc:492] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b: No bootstrap required, opened a new log
I20260620 23:05:36.235373 8920 tablet_bootstrap.cc:492] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: No bootstrap required, opened a new log
I20260620 23:05:36.235450 8922 ts_tablet_manager.cc:1397] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:36.235450 8920 ts_tablet_manager.cc:1397] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 23:05:36.237484 8920 raft_consensus.cc:348] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.237484 8922 raft_consensus.cc:348] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.237674 8920 raft_consensus.cc:374] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.237674 8922 raft_consensus.cc:374] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.237715 8920 raft_consensus.cc:729] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d9dd01f9b84b559ee4cd2dc76086a2, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.237715 8922 raft_consensus.cc:729] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96ac9253278449b8741d53f57865f8b, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.237834 8922 consensus_queue.cc:260] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.237834 8920 consensus_queue.cc:260] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.238153 8920 ts_tablet_manager.cc:1428] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:05:36.238166 8922 ts_tablet_manager.cc:1428] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:36.238204 8635 heartbeater.cc:503] Master 127.21.66.126:44103 was elected leader, sending a full tablet report...
I20260620 23:05:36.238292 8769 heartbeater.cc:503] Master 127.21.66.126:44103 was elected leader, sending a full tablet report...
I20260620 23:05:36.239117 8921 tablet_bootstrap.cc:492] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: No bootstrap required, opened a new log
I20260620 23:05:36.239197 8921 ts_tablet_manager.cc:1397] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 23:05:36.241484 8921 raft_consensus.cc:348] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.241632 8921 raft_consensus.cc:374] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.241653 8921 raft_consensus.cc:729] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b80ce9b38a54a8e8d282b50a0e7850f, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.241739 8921 consensus_queue.cc:260] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.241966 8921 ts_tablet_manager.cc:1428] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:05:36.242050 8903 heartbeater.cc:503] Master 127.21.66.126:44103 was elected leader, sending a full tablet report...
W20260620 23:05:36.278493 8770 tablet.cc:2369] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:36.280830 8927 raft_consensus.cc:482] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:36.280964 8927 raft_consensus.cc:504] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.281584 8927 leader_election.cc:288] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b80ce9b38a54a8e8d282b50a0e7850f (127.21.66.67:44147), f96ac9253278449b8741d53f57865f8b (127.21.66.65:42439)
I20260620 23:05:36.286074 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.281884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:52827 (local address 127.21.66.65:42439)
0620 23:05:36.281963 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.281965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.282084 (+ 119us) server_negotiation.cc:407] Connection header received
0620 23:05:36.282142 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.282143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.282193 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.282258 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.282555 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.283342 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.285052 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.285666 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.285734 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.285745 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.285960 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.286004 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:05:36.286074 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.281904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:36667 (local address 127.21.66.67:44147)
0620 23:05:36.281959 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.281962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.281975 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:05:36.282011 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.282012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.282029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.282160 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.282555 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.283341 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.285030 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.285636 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.285779 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.285787 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.285960 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.286003 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:05:36.286085 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.281774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:44147 (local address 127.21.66.66:36667)
0620 23:05:36.281835 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.281847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.281892 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.282282 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.282283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.282285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.282439 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.282442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.283478 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.283480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.284906 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.284911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.285790 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.285791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.285872 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.285883 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.285940 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.285991 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":128}
I20260620 23:05:36.286091 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.281787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42439 (local address 127.21.66.66:52827)
0620 23:05:36.282009 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.282021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.282062 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.282282 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.282283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.282285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.282439 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.282442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.283481 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.283482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.284967 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.284969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.285778 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.285779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.285872 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.285883 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.285940 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.285986 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":155,"thread_start_us":175,"threads_started":1}
I20260620 23:05:36.286495 8857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab8314362c4964b2d14f82d46dd6a2" candidate_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" is_pre_election: true
I20260620 23:05:36.286655 8857 raft_consensus.cc:2454] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d9dd01f9b84b559ee4cd2dc76086a2 in term 0.
I20260620 23:05:36.286878 8589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab8314362c4964b2d14f82d46dd6a2" candidate_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96ac9253278449b8741d53f57865f8b" is_pre_election: true
I20260620 23:05:36.286973 8656 leader_election.cc:302] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b80ce9b38a54a8e8d282b50a0e7850f, 87d9dd01f9b84b559ee4cd2dc76086a2; no voters:
I20260620 23:05:36.287007 8589 raft_consensus.cc:2454] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d9dd01f9b84b559ee4cd2dc76086a2 in term 0.
I20260620 23:05:36.287109 8927 raft_consensus.cc:2781] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:36.287184 8927 raft_consensus.cc:482] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:36.287210 8927 raft_consensus.cc:3037] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:36.287921 8927 raft_consensus.cc:504] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.288076 8927 leader_election.cc:288] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [CANDIDATE]: Term 1 election: Requested vote from peers 5b80ce9b38a54a8e8d282b50a0e7850f (127.21.66.67:44147), f96ac9253278449b8741d53f57865f8b (127.21.66.65:42439)
I20260620 23:05:36.288285 8589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab8314362c4964b2d14f82d46dd6a2" candidate_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96ac9253278449b8741d53f57865f8b"
I20260620 23:05:36.288268 8857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39ab8314362c4964b2d14f82d46dd6a2" candidate_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f"
I20260620 23:05:36.288374 8857 raft_consensus.cc:3037] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:36.288368 8589 raft_consensus.cc:3037] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:36.289283 8857 raft_consensus.cc:2454] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d9dd01f9b84b559ee4cd2dc76086a2 in term 1.
I20260620 23:05:36.289347 8589 raft_consensus.cc:2454] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d9dd01f9b84b559ee4cd2dc76086a2 in term 1.
I20260620 23:05:36.289534 8657 leader_election.cc:302] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d9dd01f9b84b559ee4cd2dc76086a2, f96ac9253278449b8741d53f57865f8b; no voters:
I20260620 23:05:36.289631 8927 raft_consensus.cc:2781] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:36.289808 8927 raft_consensus.cc:686] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 1 LEADER]: Becoming Leader. State: Replica: 87d9dd01f9b84b559ee4cd2dc76086a2, State: Running, Role: LEADER
I20260620 23:05:36.289918 8927 consensus_queue.cc:237] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } }
I20260620 23:05:36.290863 8443 catalog_manager.cc:5495] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d9dd01f9b84b559ee4cd2dc76086a2 (127.21.66.66). New cstate: current_term: 1 leader_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d9dd01f9b84b559ee4cd2dc76086a2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 42135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:36.304087 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.301421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.21.66.66:42135)
0620 23:05:36.301499 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.301501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.301527 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:05:36.301565 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.301566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.301587 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.301643 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:36.301810 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.302396 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.303238 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.303451 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.303511 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.303729 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:36.303853 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:36.303932 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.303976 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.303998 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:05:36.306366 8589 raft_consensus.cc:1261] T 39ab8314362c4964b2d14f82d46dd6a2 P f96ac9253278449b8741d53f57865f8b [term 1 FOLLOWER]: Refusing update from remote peer 87d9dd01f9b84b559ee4cd2dc76086a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:36.306361 8857 raft_consensus.cc:1261] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 1 FOLLOWER]: Refusing update from remote peer 87d9dd01f9b84b559ee4cd2dc76086a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:36.306731 8927 consensus_queue.cc:1035] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96ac9253278449b8741d53f57865f8b" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 42439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:36.307025 8930 consensus_queue.cc:1035] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b80ce9b38a54a8e8d282b50a0e7850f" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:36.310092 8934 mvcc.cc:205] Tried to move back new op lower bound from 7299058631907180544 to 7299058631844544512. Current Snapshot: MvccSnapshot[applied={T|T < 7299058631907180544}]
I20260620 23:05:36.310586 8933 mvcc.cc:205] Tried to move back new op lower bound from 7299058631907180544 to 7299058631844544512. Current Snapshot: MvccSnapshot[applied={T|T < 7299058631907180544}]
I20260620 23:05:36.313246 8444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:36.313349 8444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:36.314292 8444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc23ca4f93f344d29ccbd478a311676d: Sending DeleteTablet for 3 replicas of tablet 39ab8314362c4964b2d14f82d46dd6a2
I20260620 23:05:36.314695 8703 tablet_service.cc:1514] Processing DeleteTablet for tablet 39ab8314362c4964b2d14f82d46dd6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:36 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:48782
I20260620 23:05:36.314698 8837 tablet_service.cc:1514] Processing DeleteTablet for tablet 39ab8314362c4964b2d14f82d46dd6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:36 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:37522
I20260620 23:05:36.315026 8943 tablet_replica.cc:330] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: stopping tablet replica
I20260620 23:05:36.315078 8569 tablet_service.cc:1514] Processing DeleteTablet for tablet 39ab8314362c4964b2d14f82d46dd6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:36 UTC) from {username='kudu', principal='kudu/127.21.66.126@KRBTEST.COM'} at 127.0.0.1:45400
I20260620 23:05:36.315128 8943 raft_consensus.cc:2229] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:36.315256 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 8506
I20260620 23:05:36.315258 8943 raft_consensus.cc:2258] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:36.316847 8942 tablet_replica.cc:330] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: stopping tablet replica
I20260620 23:05:36.316993 8942 raft_consensus.cc:2229] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:36.317128 8942 raft_consensus.cc:2258] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:36.317682 8942 ts_tablet_manager.cc:1905] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:36.317701 8943 ts_tablet_manager.cc:1905] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:36.319525 8943 ts_tablet_manager.cc:1918] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:36.319685 8942 ts_tablet_manager.cc:1918] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:36.319765 8942 log.cc:1196] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/39ab8314362c4964b2d14f82d46dd6a2
I20260620 23:05:36.319859 8943 log.cc:1196] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/39ab8314362c4964b2d14f82d46dd6a2
I20260620 23:05:36.320037 8942 ts_tablet_manager.cc:1939] T 39ab8314362c4964b2d14f82d46dd6a2 P 5b80ce9b38a54a8e8d282b50a0e7850f: Deleting consensus metadata
I20260620 23:05:36.320287 8943 ts_tablet_manager.cc:1939] T 39ab8314362c4964b2d14f82d46dd6a2 P 87d9dd01f9b84b559ee4cd2dc76086a2: Deleting consensus metadata
I20260620 23:05:36.320547 8428 catalog_manager.cc:4841] TS 5b80ce9b38a54a8e8d282b50a0e7850f (127.21.66.67:44147): tablet 39ab8314362c4964b2d14f82d46dd6a2 (table test-table [id=6a2a9107e7764962bf42241e29d8a29d]) successfully deleted
I20260620 23:05:36.320742 8430 catalog_manager.cc:4841] TS 87d9dd01f9b84b559ee4cd2dc76086a2 (127.21.66.66:42135): tablet 39ab8314362c4964b2d14f82d46dd6a2 (table test-table [id=6a2a9107e7764962bf42241e29d8a29d]) successfully deleted
W20260620 23:05:36.322266 8429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:42439 (error 108)
I20260620 23:05:36.322449 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 8640
I20260620 23:05:36.324108 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.323968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:42439 (local address 127.0.0.1:45404)
0620 23:05:36.324030 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.324074 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:05:36.324259 8429 catalog_manager.cc:4584] TS f96ac9253278449b8741d53f57865f8b (127.21.66.65:42439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39ab8314362c4964b2d14f82d46dd6a2: Network error: Client connection negotiation failed: client connection to 127.21.66.65:42439: connect: Connection refused (error 111)
I20260620 23:05:36.327251 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 8774
I20260620 23:05:36.332029 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 8412
2026-06-20T23:05:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2015 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:36Z Disabled control of system clock
I20260620 23:05:36.360462 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44583
--webserver_interface=127.21.66.126
--webserver_port=0
--builtin_ntp_servers=127.21.66.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:05:36.446025 8949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:36.446300 8949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:36.446321 8949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:36.447487 8949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:36.447525 8949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:36.447537 8949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:36.447549 8949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:36.447561 8949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:36.448930 8949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:44583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:36.449218 8949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:36.449705 8949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:36.455112 8955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.455142 8954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.455170 8957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.455654 8949 server_base.cc:1034] running on GCE node
I20260620 23:05:36.455950 8949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:36.456235 8949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:36.457399 8949 hybrid_clock.cc:648] HybridClock initialized: now 1781996736457382 us; error 31 us; skew 500 ppm
I20260620 23:05:36.458879 8949 webserver.cc:460] Webserver started at http://127.21.66.126:44419/ using document root <none> and password file <none>
I20260620 23:05:36.459167 8949 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:36.459219 8949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:36.459363 8949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:36.460513 8949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "f3e964dae03594bbe12ff1398d5f5cdc"
server_key_iv: "d6244daf44d5bc9dad9bb6b0ac9d65f3"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.460896 8949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "f3e964dae03594bbe12ff1398d5f5cdc"
server_key_iv: "d6244daf44d5bc9dad9bb6b0ac9d65f3"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.462397 8949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:05:36.463348 8963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.463591 8949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:36.463776 8949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "f3e964dae03594bbe12ff1398d5f5cdc"
server_key_iv: "d6244daf44d5bc9dad9bb6b0ac9d65f3"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.463867 8949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.477334 8949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.477705 8949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.477865 8949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.482419 8949 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:44583
I20260620 23:05:36.482467 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:44583 every 8 connection(s)
I20260620 23:05:36.482838 8949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:36.483832 9016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.485618 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 8949
I20260620 23:05:36.485739 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:36.485941 21769 external_mini_cluster.cc:1428] Setting key d9c34ef0ca1fbe91cb05db13a77576f6
I20260620 23:05:36.486588 9016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06: Bootstrap starting.
I20260620 23:05:36.487188 9016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.487566 9016 log.cc:824] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.488294 9016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06: No bootstrap required, opened a new log
I20260620 23:05:36.490657 9016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44583 } }
I20260620 23:05:36.490845 9016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.490919 9016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc4c2d4f18d42a4bd24fd1fa05beb06, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.491055 9016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44583 } }
I20260620 23:05:36.491151 9016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:36.491187 9016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:36.491240 9016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:36.491907 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.486530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.21.66.126:44583)
0620 23:05:36.486738 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.486746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.486771 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:05:36.487185 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.487192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.487253 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.487415 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.487755 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.488783 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.489868 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.490331 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.490424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.491201 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.491206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.491206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:36.491427 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.491463 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:36.491474 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.491528 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.491633 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.491694 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260620 23:05:36.491958 9016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44583 } }
I20260620 23:05:36.492074 9016 leader_election.cc:302] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cc4c2d4f18d42a4bd24fd1fa05beb06; no voters:
I20260620 23:05:36.492231 9016 leader_election.cc:288] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:36.492331 9021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:36.492488 9021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc4c2d4f18d42a4bd24fd1fa05beb06, State: Running, Role: LEADER
I20260620 23:05:36.492542 9016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:36.492583 9021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44583 } }
I20260620 23:05:36.492966 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cc4c2d4f18d42a4bd24fd1fa05beb06. Latest consensus state: current_term: 1 leader_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44583 } } }
I20260620 23:05:36.492973 9023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc4c2d4f18d42a4bd24fd1fa05beb06" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 44583 } } }
I20260620 23:05:36.493085 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:36.493094 9023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:36.493381 9029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:36.493889 9029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:36.495476 9029 catalog_manager.cc:1347] Generated new cluster ID: a037c54790a24a4a98ccb2fd2a07a12b
I20260620 23:05:36.495544 9029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:36.504335 9029 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:36.504914 9029 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:36.514991 9029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06: Generated new TSK 0
I20260620 23:05:36.515302 9029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:36.523849 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.65:0
--local_ip_for_outbound_sockets=127.21.66.65
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44583
--builtin_ntp_servers=127.21.66.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:05:36.607156 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:36.607407 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:36.607445 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:36.608713 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:36.608774 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:36.608816 9040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.65
I20260620 23:05:36.610208 9040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.66.65
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.65
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:36.610473 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:36.610780 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:36.616701 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.616717 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.616799 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.616964 9040 server_base.cc:1034] running on GCE node
I20260620 23:05:36.617175 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:36.617479 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:36.618652 9040 hybrid_clock.cc:648] HybridClock initialized: now 1781996736618627 us; error 36 us; skew 500 ppm
I20260620 23:05:36.620241 9040 webserver.cc:460] Webserver started at http://127.21.66.65:44233/ using document root <none> and password file <none>
I20260620 23:05:36.620505 9040 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:36.620568 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:36.620703 9040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:36.622072 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data/instance:
uuid: "47aa9f9a8e9242c8adb3d0803706beb7"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "3c2a46431ab6671678acc1cb93cd5b93"
server_key_iv: "3569ca1239a9aa9e88706c9a9fd61635"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.622538 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance:
uuid: "47aa9f9a8e9242c8adb3d0803706beb7"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "3c2a46431ab6671678acc1cb93cd5b93"
server_key_iv: "3569ca1239a9aa9e88706c9a9fd61635"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.624400 9040 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:36.625649 9054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.625861 9040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:36.625983 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/wal
uuid: "47aa9f9a8e9242c8adb3d0803706beb7"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "3c2a46431ab6671678acc1cb93cd5b93"
server_key_iv: "3569ca1239a9aa9e88706c9a9fd61635"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.626075 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.647044 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.647425 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.647603 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.647881 9040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:36.648284 9040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:36.648345 9040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.648387 9040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:36.648416 9040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.654268 9040 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.65:40951
I20260620 23:05:36.654322 9167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.65:40951 every 8 connection(s)
I20260620 23:05:36.654702 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/data/info.pb
I20260620 23:05:36.659552 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9040
I20260620 23:05:36.659683 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-0/wal/instance
I20260620 23:05:36.659897 21769 external_mini_cluster.cc:1428] Setting key 16006c69309c4d3c5286ebe1b9e771b9
I20260620 23:05:36.660223 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.655529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.65:44977 (local address 127.21.66.126:44583)
0620 23:05:36.655597 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.655599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.655879 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:05:36.656400 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.656402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.656417 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.656473 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.657146 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.657660 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.658662 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.658868 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.658958 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.659903 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.659905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.659905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:36.659910 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.659926 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:36.659931 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.659979 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.660118 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.660144 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:05:36.660473 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.655527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44583 (local address 127.21.66.65:44977)
0620 23:05:36.655701 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.655782 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.656295 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.656543 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.656546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.656554 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:36.657051 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.657055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.657765 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.657766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.658570 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.658572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.658970 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.658971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.659104 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.659719 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:36.659719 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:36.659772 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:36.659801 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:36.660014 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:36.660019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:36.660022 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.660106 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.660209 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260620 23:05:36.660965 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.66:0
--local_ip_for_outbound_sockets=127.21.66.66
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44583
--builtin_ntp_servers=127.21.66.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:05:36.661146 9168 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44583
I20260620 23:05:36.661248 9168 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:36.661450 9168 heartbeater.cc:511] Master 127.21.66.126:44583 requested a full tablet report, sending...
I20260620 23:05:36.661933 8980 ts_manager.cc:194] Registered new tserver with Master: 47aa9f9a8e9242c8adb3d0803706beb7 (127.21.66.65:40951)
I20260620 23:05:36.662626 8980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.65:44977
W20260620 23:05:36.743849 9171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:36.744056 9171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:36.744078 9171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:36.745257 9171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:36.745302 9171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:36.745327 9171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.66
I20260620 23:05:36.746709 9171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.66.66
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.66
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:36.746969 9171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:36.747284 9171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:36.752579 9177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.752538 9179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.752553 9176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.752960 9171 server_base.cc:1034] running on GCE node
I20260620 23:05:36.753218 9171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:36.753496 9171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:36.754660 9171 hybrid_clock.cc:648] HybridClock initialized: now 1781996736754632 us; error 41 us; skew 500 ppm
I20260620 23:05:36.756040 9171 webserver.cc:460] Webserver started at http://127.21.66.66:35109/ using document root <none> and password file <none>
I20260620 23:05:36.756320 9171 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:36.756371 9171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:36.756505 9171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:36.757614 9171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data/instance:
uuid: "0168fba90b084ae3ad641284ca2644c2"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "016d7da7f80ffdd441ade468b89b6b7b"
server_key_iv: "a370b91be5ead761c04101fc1b060773"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.757978 9171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance:
uuid: "0168fba90b084ae3ad641284ca2644c2"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "016d7da7f80ffdd441ade468b89b6b7b"
server_key_iv: "a370b91be5ead761c04101fc1b060773"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.759377 9171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:36.760226 9185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.760452 9171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:36.760545 9171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal
uuid: "0168fba90b084ae3ad641284ca2644c2"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "016d7da7f80ffdd441ade468b89b6b7b"
server_key_iv: "a370b91be5ead761c04101fc1b060773"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.760622 9171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.777025 9171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.777426 9171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.777634 9171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.777933 9171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:36.778389 9171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:36.778446 9171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.778481 9171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:36.778507 9171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.784677 9171 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.66:46721
I20260620 23:05:36.784715 9298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.66:46721 every 8 connection(s)
I20260620 23:05:36.785054 9171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/data/info.pb
I20260620 23:05:36.786490 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9171
I20260620 23:05:36.786616 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal/instance
I20260620 23:05:36.786847 21769 external_mini_cluster.cc:1428] Setting key 2b47578dd225d7fe6b87ce4292b14151
I20260620 23:05:36.788017 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.66.67:0
--local_ip_for_outbound_sockets=127.21.66.67
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44583
--builtin_ntp_servers=127.21.66.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:05:36.791464 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.785852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.66:49549 (local address 127.21.66.126:44583)
0620 23:05:36.785936 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.785938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.786175 (+ 237us) server_negotiation.cc:407] Connection header received
0620 23:05:36.786904 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.786905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.786919 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.786997 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.787889 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.788565 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.789896 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.790089 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.790228 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.791154 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.791156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.791156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:36.791161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.791178 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:36.791182 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.791252 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.791353 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.791383 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:05:36.791692 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.785812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44583 (local address 127.21.66.66:49549)
0620 23:05:36.786003 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.786045 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.786780 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.787081 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.787085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.787094 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:36.787762 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.787770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.788716 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.788718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.789789 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.789792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.790242 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.790243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.790388 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.790993 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:36.790993 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:36.791023 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:36.791047 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:36.791252 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:36.791259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:36.791262 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.791356 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.791461 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260620 23:05:36.792124 9299 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44583
I20260620 23:05:36.792243 9299 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:36.792455 9299 heartbeater.cc:511] Master 127.21.66.126:44583 requested a full tablet report, sending...
I20260620 23:05:36.792843 8980 ts_manager.cc:194] Registered new tserver with Master: 0168fba90b084ae3ad641284ca2644c2 (127.21.66.66:46721)
I20260620 23:05:36.793215 8980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.66:49549
W20260620 23:05:36.880268 9302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:36.880545 9302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:36.880585 9302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:36.881922 9302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:36.881999 9302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:36.882050 9302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.66.67
I20260620 23:05:36.883807 9302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.66.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.66.67
--webserver_port=0
--tserver_master_addrs=127.21.66.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.66.67
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:36.884124 9302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:36.884467 9302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:36.890148 9310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:36.890202 9307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.890233 9302 server_base.cc:1034] running on GCE node
W20260620 23:05:36.890174 9308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:36.890539 9302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:36.890779 9302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:36.891953 9302 hybrid_clock.cc:648] HybridClock initialized: now 1781996736891935 us; error 41 us; skew 500 ppm
I20260620 23:05:36.893353 9302 webserver.cc:460] Webserver started at http://127.21.66.67:46555/ using document root <none> and password file <none>
I20260620 23:05:36.893656 9302 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:36.893733 9302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:36.893894 9302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:36.895080 9302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data/instance:
uuid: "e07e7e46d089498ea4d74b5edcc102d3"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "6dec69bb05b03b027c067ab73acff39e"
server_key_iv: "f8dc0b6c0bbadd2084f8d6af4c474117"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.895453 9302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance:
uuid: "e07e7e46d089498ea4d74b5edcc102d3"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "6dec69bb05b03b027c067ab73acff39e"
server_key_iv: "f8dc0b6c0bbadd2084f8d6af4c474117"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.896862 9302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:05:36.897717 9316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.897939 9302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:36.898078 9302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal
uuid: "e07e7e46d089498ea4d74b5edcc102d3"
format_stamp: "Formatted at 2026-06-20 23:05:36 on dist-test-slave-qgtk"
server_key: "6dec69bb05b03b027c067ab73acff39e"
server_key_iv: "f8dc0b6c0bbadd2084f8d6af4c474117"
server_key_version: "encryptionkey@0"
I20260620 23:05:36.898198 9302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:36.913868 9302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:36.914310 9302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:36.914481 9302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:36.914738 9302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:05:36.915092 9302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:05:36.915146 9302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.915181 9302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:05:36.915210 9302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:36.920886 9302 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.67:36187
I20260620 23:05:36.920966 9429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.67:36187 every 8 connection(s)
I20260620 23:05:36.921308 9302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/data/info.pb
I20260620 23:05:36.923552 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9302
I20260620 23:05:36.923704 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal/instance
I20260620 23:05:36.923924 21769 external_mini_cluster.cc:1428] Setting key 47c643912f9a1128562c509d10e5d9b4
I20260620 23:05:36.927347 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.922157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:54251 (local address 127.21.66.126:44583)
0620 23:05:36.922220 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.922223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.922412 (+ 189us) server_negotiation.cc:407] Connection header received
0620 23:05:36.923036 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.923038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.923050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.923123 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.923939 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.924719 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.925809 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.926027 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.926150 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.927031 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.927033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.927033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:36.927038 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.927054 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:36.927058 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.927139 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.927240 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.927267 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 23:05:36.927582 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.922068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:44583 (local address 127.21.66.67:54251)
0620 23:05:36.922292 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.922327 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.922922 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.923198 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.923202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.923209 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:05:36.923832 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.923837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.924803 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.924804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.925722 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.925725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.926150 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.926151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.926294 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.926880 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:05:36.926881 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:05:36.926909 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:05:36.926933 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:05:36.927143 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:05:36.927149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:05:36.927152 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.927239 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.927343 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260620 23:05:36.928085 9430 heartbeater.cc:347] Connected to a master server at 127.21.66.126:44583
I20260620 23:05:36.928186 9430 heartbeater.cc:464] Registering TS with master...
I20260620 23:05:36.928393 9430 heartbeater.cc:511] Master 127.21.66.126:44583 requested a full tablet report, sending...
I20260620 23:05:36.928828 8980 ts_manager.cc:194] Registered new tserver with Master: e07e7e46d089498ea4d74b5edcc102d3 (127.21.66.67:36187)
I20260620 23:05:36.929220 8980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.66.67:54251
I20260620 23:05:36.936197 21769 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:05:36.941287 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.937800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.21.66.126:44583)
0620 23:05:36.937884 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.937886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.937985 (+ 99us) server_negotiation.cc:407] Connection header received
0620 23:05:36.938064 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.938065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.938078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.938172 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:36.938380 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.938973 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.939859 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.940093 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.940191 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.940986 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:36.940988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:36.940989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:36.940994 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:36.941012 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:36.941017 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:36.941075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.941147 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.941174 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 23:05:36.943307 8980 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:05:36.944916 8980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:05:36.951777 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.947429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:40951 (local address 127.0.0.1:40496)
0620 23:05:36.947851 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.947862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.947900 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.948264 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.948265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.948267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.948299 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.948301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.949637 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.949638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.950669 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.950672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.951511 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.951512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.951574 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.951588 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.951650 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.951693 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":107,"threads_started":1}
I20260620 23:05:36.951813 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.947900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40496 (local address 127.21.66.65:40951)
0620 23:05:36.948091 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.948095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.948104 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:05:36.948150 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.948152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.948165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.948367 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.948422 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.949569 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.950762 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.951425 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.951502 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.951594 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.951681 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.951722 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
I20260620 23:05:36.952847 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.947415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.67:36187 (local address 127.0.0.1:57728)
0620 23:05:36.947608 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.947653 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.947725 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.948163 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.948164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.948167 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.948236 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.948238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.949404 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.949405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.950983 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.950986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.952559 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.952560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.952629 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.952634 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.952707 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.952758 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":36,"thread_start_us":76,"threads_started":1}
I20260620 23:05:36.952769 9102 tablet_service.cc:1467] Processing CreateTablet for tablet f721fa003c0d428c914993bd848677be (DEFAULT_TABLE table=test-table [id=7ef2b02db75d48e98d1919325dbfe432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:36.953135 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.947692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.21.66.66:46721)
0620 23:05:36.947888 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.947893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.947905 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:05:36.947946 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.947948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.947965 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.948254 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.948312 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.949484 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.951082 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.951815 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.952880 (+ 1065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.953009 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.953037 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.953076 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260620 23:05:36.953265 9102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f721fa003c0d428c914993bd848677be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.953357 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.947608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.21.66.67:36187)
0620 23:05:36.947808 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:36.947813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:36.947825 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:05:36.947879 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:36.947882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:36.947900 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:36.948087 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:36.948326 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.949291 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.951912 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.952467 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.952538 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.952646 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:36.953142 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 23:05:36.953194 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260620 23:05:36.952975 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:36.947416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:46721 (local address 127.0.0.1:44480)
0620 23:05:36.947608 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:36.947661 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:36.947724 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:36.948083 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:36.948086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:36.948090 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:36.948173 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.948176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.949581 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.949582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.950983 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:36.950986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:36.952710 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:36.952710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:36.952767 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:36.952771 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:36.952796 (+ 25us) client_negotiation.cc:240] Negotiation successful
0620 23:05:36.952829 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":70,"thread_start_us":94,"threads_started":1}
I20260620 23:05:36.954202 9364 tablet_service.cc:1467] Processing CreateTablet for tablet f721fa003c0d428c914993bd848677be (DEFAULT_TABLE table=test-table [id=7ef2b02db75d48e98d1919325dbfe432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:36.954530 9364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f721fa003c0d428c914993bd848677be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.956712 9444 tablet_bootstrap.cc:492] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7: Bootstrap starting.
I20260620 23:05:36.957442 9444 tablet_bootstrap.cc:654] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.957674 9445 tablet_bootstrap.cc:492] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Bootstrap starting.
I20260620 23:05:36.957778 9444 log.cc:824] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.958154 9233 tablet_service.cc:1467] Processing CreateTablet for tablet f721fa003c0d428c914993bd848677be (DEFAULT_TABLE table=test-table [id=7ef2b02db75d48e98d1919325dbfe432]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:05:36.958437 9445 tablet_bootstrap.cc:654] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.958498 9233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f721fa003c0d428c914993bd848677be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:36.958696 9444 tablet_bootstrap.cc:492] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7: No bootstrap required, opened a new log
I20260620 23:05:36.958807 9445 log.cc:824] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.958781 9444 ts_tablet_manager.cc:1397] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:36.959532 9445 tablet_bootstrap.cc:492] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: No bootstrap required, opened a new log
I20260620 23:05:36.959621 9445 ts_tablet_manager.cc:1397] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:36.961602 9445 raft_consensus.cc:348] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:36.961776 9445 raft_consensus.cc:374] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.961669 9444 raft_consensus.cc:348] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:36.961825 9445 raft_consensus.cc:729] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07e7e46d089498ea4d74b5edcc102d3, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.961831 9444 raft_consensus.cc:374] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.961870 9444 raft_consensus.cc:729] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47aa9f9a8e9242c8adb3d0803706beb7, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.961951 9445 consensus_queue.cc:260] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:36.961987 9444 consensus_queue.cc:260] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:36.962262 9444 ts_tablet_manager.cc:1428] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:05:36.962282 9445 ts_tablet_manager.cc:1428] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 23:05:36.962395 9168 heartbeater.cc:503] Master 127.21.66.126:44583 was elected leader, sending a full tablet report...
I20260620 23:05:36.962424 9430 heartbeater.cc:503] Master 127.21.66.126:44583 was elected leader, sending a full tablet report...
I20260620 23:05:36.962478 9448 tablet_bootstrap.cc:492] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Bootstrap starting.
I20260620 23:05:36.963152 9448 tablet_bootstrap.cc:654] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:36.963490 9448 log.cc:824] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:36.964205 9448 tablet_bootstrap.cc:492] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: No bootstrap required, opened a new log
I20260620 23:05:36.964257 9448 ts_tablet_manager.cc:1397] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:36.966046 9448 raft_consensus.cc:348] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:36.966243 9448 raft_consensus.cc:374] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:36.966269 9448 raft_consensus.cc:729] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0168fba90b084ae3ad641284ca2644c2, State: Initialized, Role: FOLLOWER
I20260620 23:05:36.966408 9448 consensus_queue.cc:260] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:36.966693 9448 ts_tablet_manager.cc:1428] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:05:36.966782 9299 heartbeater.cc:503] Master 127.21.66.126:44583 was elected leader, sending a full tablet report...
W20260620 23:05:37.035619 9300 tablet.cc:2369] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:05:37.059579 9450 raft_consensus.cc:482] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:05:37.059798 9450 raft_consensus.cc:504] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:37.060226 9450 leader_election.cc:288] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0168fba90b084ae3ad641284ca2644c2 (127.21.66.66:46721), 47aa9f9a8e9242c8adb3d0803706beb7 (127.21.66.65:40951)
I20260620 23:05:37.063644 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.060438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:40951 (local address 127.21.66.67:55215)
0620 23:05:37.060516 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:37.060529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:37.060580 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:37.060783 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:37.060784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:37.060785 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:37.060882 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:37.060884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.061641 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.061643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:37.062700 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:37.062702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.063363 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.063364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:37.063437 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.063440 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 23:05:37.063518 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 23:05:37.063563 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:05:37.063644 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.060507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:55215 (local address 127.21.66.65:40951)
0620 23:05:37.060588 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:37.060590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:37.060603 (+ 13us) server_negotiation.cc:407] Connection header received
0620 23:05:37.060688 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:37.060689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:37.060703 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:37.060773 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:37.060981 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.061542 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.062809 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.063264 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.063363 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.063373 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:37.063529 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:05:37.063568 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:05:37.063956 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.060504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.66.67:56541 (local address 127.21.66.66:46721)
0620 23:05:37.060585 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:37.060588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:37.060828 (+ 240us) server_negotiation.cc:407] Connection header received
0620 23:05:37.060882 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:37.060883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:37.060896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:37.060954 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:05:37.061164 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.061832 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.062904 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.063353 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.063459 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.063467 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:37.063769 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:05:37.063855 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:05:37.064071 9122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f721fa003c0d428c914993bd848677be" candidate_uuid: "e07e7e46d089498ea4d74b5edcc102d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" is_pre_election: true
I20260620 23:05:37.064234 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.060444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.66:46721 (local address 127.21.66.67:56541)
0620 23:05:37.060741 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:37.060752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:05:37.060814 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:05:37.060975 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:05:37.060976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:05:37.060978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:05:37.061043 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:37.061044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.061945 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.061946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:37.062801 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:05:37.062812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.063457 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.063458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:05:37.063568 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.063572 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:05:37.064060 (+ 488us) client_negotiation.cc:240] Negotiation successful
0620 23:05:37.064122 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":93,"thread_start_us":120,"threads_started":1}
I20260620 23:05:37.064260 9122 raft_consensus.cc:2454] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07e7e46d089498ea4d74b5edcc102d3 in term 0.
I20260620 23:05:37.064622 9253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f721fa003c0d428c914993bd848677be" candidate_uuid: "e07e7e46d089498ea4d74b5edcc102d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0168fba90b084ae3ad641284ca2644c2" is_pre_election: true
I20260620 23:05:37.064674 9319 leader_election.cc:302] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47aa9f9a8e9242c8adb3d0803706beb7, e07e7e46d089498ea4d74b5edcc102d3; no voters:
I20260620 23:05:37.064792 9253 raft_consensus.cc:2454] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07e7e46d089498ea4d74b5edcc102d3 in term 0.
I20260620 23:05:37.064823 9450 raft_consensus.cc:2781] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:05:37.064900 9450 raft_consensus.cc:482] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:05:37.064925 9450 raft_consensus.cc:3037] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:37.065686 9450 raft_consensus.cc:504] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:37.065841 9450 leader_election.cc:288] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [CANDIDATE]: Term 1 election: Requested vote from peers 0168fba90b084ae3ad641284ca2644c2 (127.21.66.66:46721), 47aa9f9a8e9242c8adb3d0803706beb7 (127.21.66.65:40951)
I20260620 23:05:37.066025 9253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f721fa003c0d428c914993bd848677be" candidate_uuid: "e07e7e46d089498ea4d74b5edcc102d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0168fba90b084ae3ad641284ca2644c2"
I20260620 23:05:37.066093 9122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f721fa003c0d428c914993bd848677be" candidate_uuid: "e07e7e46d089498ea4d74b5edcc102d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47aa9f9a8e9242c8adb3d0803706beb7"
I20260620 23:05:37.066141 9253 raft_consensus.cc:3037] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:37.066224 9122 raft_consensus.cc:3037] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:37.067080 9122 raft_consensus.cc:2454] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07e7e46d089498ea4d74b5edcc102d3 in term 1.
I20260620 23:05:37.067332 9253 raft_consensus.cc:2454] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07e7e46d089498ea4d74b5edcc102d3 in term 1.
I20260620 23:05:37.067365 9319 leader_election.cc:302] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47aa9f9a8e9242c8adb3d0803706beb7, e07e7e46d089498ea4d74b5edcc102d3; no voters:
I20260620 23:05:37.067472 9450 raft_consensus.cc:2781] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:37.067642 9450 raft_consensus.cc:686] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 1 LEADER]: Becoming Leader. State: Replica: e07e7e46d089498ea4d74b5edcc102d3, State: Running, Role: LEADER
I20260620 23:05:37.067973 9450 consensus_queue.cc:237] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } }
I20260620 23:05:37.069167 8980 catalog_manager.cc:5495] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07e7e46d089498ea4d74b5edcc102d3 (127.21.66.67). New cstate: current_term: 1 leader_uuid: "e07e7e46d089498ea4d74b5edcc102d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e07e7e46d089498ea4d74b5edcc102d3" member_type: VOTER last_known_addr { host: "127.21.66.67" port: 36187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 } health_report { overall_health: UNKNOWN } } }
I20260620 23:05:37.099187 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.096169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.21.66.67:36187)
0620 23:05:37.096278 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:37.096281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:37.096290 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:05:37.096330 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:37.096331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:37.096350 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:37.096432 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:05:37.096657 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.097177 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.098101 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:37.098340 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:37.098411 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:37.098723 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:05:37.098837 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:05:37.098912 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:37.099023 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 23:05:37.099042 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 23:05:37.101404 9122 raft_consensus.cc:1261] T f721fa003c0d428c914993bd848677be P 47aa9f9a8e9242c8adb3d0803706beb7 [term 1 FOLLOWER]: Refusing update from remote peer e07e7e46d089498ea4d74b5edcc102d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:37.101405 9253 raft_consensus.cc:1261] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 1 FOLLOWER]: Refusing update from remote peer e07e7e46d089498ea4d74b5edcc102d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:05:37.101840 9450 consensus_queue.cc:1035] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0168fba90b084ae3ad641284ca2644c2" member_type: VOTER last_known_addr { host: "127.21.66.66" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:37.102008 9454 consensus_queue.cc:1035] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47aa9f9a8e9242c8adb3d0803706beb7" member_type: VOTER last_known_addr { host: "127.21.66.65" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:05:37.105201 9457 mvcc.cc:205] Tried to move back new op lower bound from 7299058635164372992 to 7299058635031928832. Current Snapshot: MvccSnapshot[applied={T|T < 7299058635164372992 or (T in {7299058635164372992})}]
I20260620 23:05:37.105293 9458 mvcc.cc:205] Tried to move back new op lower bound from 7299058635164372992 to 7299058635031928832. Current Snapshot: MvccSnapshot[applied={T|T < 7299058635164372992}]
I20260620 23:05:37.107129 8980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:37.107231 8980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:05:37.107988 8980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cc4c2d4f18d42a4bd24fd1fa05beb06: Sending DeleteTablet for 3 replicas of tablet f721fa003c0d428c914993bd848677be
I20260620 23:05:37.108402 9102 tablet_service.cc:1514] Processing DeleteTablet for tablet f721fa003c0d428c914993bd848677be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:37 UTC) from {username='slave'} at 127.0.0.1:40496
I20260620 23:05:37.108404 9364 tablet_service.cc:1514] Processing DeleteTablet for tablet f721fa003c0d428c914993bd848677be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:37 UTC) from {username='slave'} at 127.0.0.1:57728
I20260620 23:05:37.108827 9233 tablet_service.cc:1514] Processing DeleteTablet for tablet f721fa003c0d428c914993bd848677be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:05:37 UTC) from {username='slave'} at 127.0.0.1:44480
I20260620 23:05:37.109115 9468 tablet_replica.cc:330] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: stopping tablet replica
I20260620 23:05:37.109326 9468 raft_consensus.cc:2229] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:05:37.109386 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9040
I20260620 23:05:37.109462 9468 raft_consensus.cc:2258] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:37.111106 9468 ts_tablet_manager.cc:1905] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:37.113193 9468 ts_tablet_manager.cc:1918] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:37.113283 9468 log.cc:1196] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-1/wal/wals/f721fa003c0d428c914993bd848677be
I20260620 23:05:37.113528 9468 ts_tablet_manager.cc:1939] T f721fa003c0d428c914993bd848677be P 0168fba90b084ae3ad641284ca2644c2: Deleting consensus metadata
I20260620 23:05:37.114172 8964 catalog_manager.cc:4841] TS 0168fba90b084ae3ad641284ca2644c2 (127.21.66.66:46721): tablet f721fa003c0d428c914993bd848677be (table test-table [id=7ef2b02db75d48e98d1919325dbfe432]) successfully deleted
W20260620 23:05:37.115027 8966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.66.65:40951 (error 108)
I20260620 23:05:37.115236 9467 tablet_replica.cc:330] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: stopping tablet replica
I20260620 23:05:37.115389 9467 raft_consensus.cc:2229] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:05:37.115535 9467 raft_consensus.cc:2258] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:05:37.115753 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9171
I20260620 23:05:37.116061 9467 ts_tablet_manager.cc:1905] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:05:37.118089 9467 ts_tablet_manager.cc:1918] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:05:37.118184 9467 log.cc:1196] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Deleting WAL directory at /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781996487602640-21769-0/minicluster-data/ts-2/wal/wals/f721fa003c0d428c914993bd848677be
I20260620 23:05:37.118431 9467 ts_tablet_manager.cc:1939] T f721fa003c0d428c914993bd848677be P e07e7e46d089498ea4d74b5edcc102d3: Deleting consensus metadata
I20260620 23:05:37.118871 8967 catalog_manager.cc:4841] TS e07e7e46d089498ea4d74b5edcc102d3 (127.21.66.67:36187): tablet f721fa003c0d428c914993bd848677be (table test-table [id=7ef2b02db75d48e98d1919325dbfe432]) successfully deleted
I20260620 23:05:37.122711 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:37.122499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.65:40951 (local address 127.0.0.1:40502)
0620 23:05:37.122562 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 23:05:37.122612 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.65:40951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 23:05:37.122921 8966 catalog_manager.cc:4584] TS 47aa9f9a8e9242c8adb3d0803706beb7 (127.21.66.65:40951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f721fa003c0d428c914993bd848677be: Network error: Client connection negotiation failed: client connection to 127.21.66.65:40951: connect: Connection refused (error 111)
I20260620 23:05:37.123345 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9302
I20260620 23:05:37.128152 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 8949
2026-06-20T23:05:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (794 ms)
[----------] 6 tests from EncryptionPolicyTest (8607 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:37 dist-test-slave-qgtk krb5kdc[9473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:37 dist-test-slave-qgtk krb5kdc[9473](info): set up 2 sockets
Jun 20 23:05:37 dist-test-slave-qgtk krb5kdc[9473](info): commencing operation
krb5kdc: starting...
W20260620 23:05:38.251453 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:38 dist-test-slave-qgtk krb5kdc[9473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:38.382026 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:38.472564 9486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:38.472890 9486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:38.472941 9486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:38.474453 9486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:38.474542 9486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:38.474567 9486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:38.474591 9486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:38.474623 9486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:38.476502 9486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:38.476998 9486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:38.477312 9486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:38.482829 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:38.482833 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:38.482896 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:38.483058 9486 server_base.cc:1034] running on GCE node
I20260620 23:05:38.483381 9486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:38.483693 9486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:38.484920 9486 hybrid_clock.cc:648] HybridClock initialized: now 1781996738484888 us; error 41 us; skew 500 ppm
Jun 20 23:05:38 dist-test-slave-qgtk krb5kdc[9473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:38.487718 9486 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:38.488242 9486 webserver.cc:460] Webserver started at http://127.0.0.1:39733/ using document root <none> and password file <none>
I20260620 23:05:38.488490 9486 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:38.488562 9486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:38.488713 9486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:38.489982 9486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "7ccbd11cfdca42d3834f868ad690cb6d"
format_stamp: "Formatted at 2026-06-20 23:05:38 on dist-test-slave-qgtk"
server_key: "d11fd59d81ecad6ba268925699d6b7d6"
server_key_iv: "04cf39ec39484a143ee627c9f65a4306"
server_key_version: "encryptionkey@0"
I20260620 23:05:38.490504 9486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "7ccbd11cfdca42d3834f868ad690cb6d"
format_stamp: "Formatted at 2026-06-20 23:05:38 on dist-test-slave-qgtk"
server_key: "d11fd59d81ecad6ba268925699d6b7d6"
server_key_iv: "04cf39ec39484a143ee627c9f65a4306"
server_key_version: "encryptionkey@0"
I20260620 23:05:38.492090 9486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:38.493044 9501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:38.493263 9486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:38.493418 9486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "7ccbd11cfdca42d3834f868ad690cb6d"
format_stamp: "Formatted at 2026-06-20 23:05:38 on dist-test-slave-qgtk"
server_key: "d11fd59d81ecad6ba268925699d6b7d6"
server_key_iv: "04cf39ec39484a143ee627c9f65a4306"
server_key_version: "encryptionkey@0"
I20260620 23:05:38.493515 9486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:38.509999 9486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:38.512305 9486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:38.512632 9486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:38.517426 9486 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45355
I20260620 23:05:38.517544 9553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45355 every 8 connection(s)
I20260620 23:05:38.517882 9486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:38.518851 9554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:38.521256 9554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d: Bootstrap starting.
I20260620 23:05:38.521878 9554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:38.522254 9554 log.cc:824] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:38.523108 9554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d: No bootstrap required, opened a new log
I20260620 23:05:38.524910 9554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260620 23:05:38.525070 9554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:38.525094 9554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccbd11cfdca42d3834f868ad690cb6d, State: Initialized, Role: FOLLOWER
I20260620 23:05:38.525269 9554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260620 23:05:38.525349 9554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:38.525372 9554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:38.525401 9554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:38.526028 9554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260620 23:05:38.526142 9554 leader_election.cc:302] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ccbd11cfdca42d3834f868ad690cb6d; no voters:
I20260620 23:05:38.526283 9554 leader_election.cc:288] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:38.526454 9557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:38.526563 9554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:38.526647 9557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [term 1 LEADER]: Becoming Leader. State: Replica: 7ccbd11cfdca42d3834f868ad690cb6d, State: Running, Role: LEADER
I20260620 23:05:38.526746 9557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } }
I20260620 23:05:38.527163 9558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } } }
I20260620 23:05:38.527266 9558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:38.527293 9559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccbd11cfdca42d3834f868ad690cb6d. Latest consensus state: current_term: 1 leader_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ccbd11cfdca42d3834f868ad690cb6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45355 } } }
I20260620 23:05:38.527374 9559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:38.527571 9566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:38.527897 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9486
I20260620 23:05:38.528021 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:38.528136 9566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:38.528251 21769 external_mini_cluster.cc:1428] Setting key fb35ffb7abc687418842b87cb3fc9dfc
I20260620 23:05:38.529822 9566 catalog_manager.cc:1347] Generated new cluster ID: 27abacd2b4e2462ab4c732cfd277c040
I20260620 23:05:38.529879 9566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:05:38 dist-test-slave-qgtk krb5kdc[9473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:05:38.539408 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:38.528966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48083 (local address 127.0.0.1:45355)
0620 23:05:38.529127 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:38.529137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:38.529170 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:05:38.529322 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:38.529330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:38.529393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:38.529543 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:38.530485 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:38.531671 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:38.532533 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:38.533006 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:38.533132 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:38.535762 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:38.535766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:38.535767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:38.535787 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:38.537628 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:38.538050 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:38.538061 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:38.538061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:38.538224 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:38.538471 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:38.538472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:38.538473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:38.538729 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:38.538924 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:38.539013 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:05:38.539225 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260620 23:05:38.540562 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:38.548345 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:38.541928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51057 (local address 127.0.0.1:45355)
0620 23:05:38.541968 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:38.541971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:38.542111 (+ 140us) server_negotiation.cc:407] Connection header received
0620 23:05:38.542181 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:38.542182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:38.542192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:38.542248 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:38.542754 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:38.543275 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:38.544004 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:38.544327 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:38.544422 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:38.545603 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:38.545604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:38.545605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:38.545612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:38.547096 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:38.547469 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:38.547471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:38.547471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:38.547531 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:38.547784 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:38.547786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:38.547786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:38.547904 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:38.548030 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:38.548150 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 23:05:38.548213 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:05:38.549755 9566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:38.550501 9566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:38.561883 9566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ccbd11cfdca42d3834f868ad690cb6d: Generated new TSK 0
I20260620 23:05:38.562180 9566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:38.567154 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9486
W20260620 23:05:38.571092 9471 connection.cc:535] client connection to 127.0.0.1:45355 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45355): Connection reset by peer (error 104)
2026-06-20T23:05:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:38 dist-test-slave-qgtk krb5kdc[9585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:38 dist-test-slave-qgtk krb5kdc[9585](info): set up 2 sockets
Jun 20 23:05:38 dist-test-slave-qgtk krb5kdc[9585](info): commencing operation
krb5kdc: starting...
W20260620 23:05:39.639210 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:39 dist-test-slave-qgtk krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:39.762095 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:39.847250 9598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:39.847468 9598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:39.847492 9598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:39.848613 9598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:39.848659 9598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:39.848672 9598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:39.848685 9598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:39.848697 9598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:39.850184 9598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:39.850384 9598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:39.850565 9598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:39.855645 9604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:39.855661 9603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:39.855661 9606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:39.855800 9598 server_base.cc:1034] running on GCE node
I20260620 23:05:39.856108 9598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:39.856434 9598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:39.857647 9598 hybrid_clock.cc:648] HybridClock initialized: now 1781996739857629 us; error 53 us; skew 500 ppm
Jun 20 23:05:39 dist-test-slave-qgtk krb5kdc[9585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:39.863955 9598 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:39.864522 9598 webserver.cc:460] Webserver started at http://127.0.0.1:40379/ using document root <none> and password file <none>
I20260620 23:05:39.864784 9598 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:39.864856 9598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:39.865010 9598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:39.866232 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "917900b716c54db3a390d21d31f9bad8"
format_stamp: "Formatted at 2026-06-20 23:05:39 on dist-test-slave-qgtk"
server_key: "ec80684296c9afc65608f4948770f5a4"
server_key_iv: "f381835599356fa8b7f9ebd445ba36b8"
server_key_version: "encryptionkey@0"
I20260620 23:05:39.866611 9598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "917900b716c54db3a390d21d31f9bad8"
format_stamp: "Formatted at 2026-06-20 23:05:39 on dist-test-slave-qgtk"
server_key: "ec80684296c9afc65608f4948770f5a4"
server_key_iv: "f381835599356fa8b7f9ebd445ba36b8"
server_key_version: "encryptionkey@0"
I20260620 23:05:39.868124 9598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:05:39.868948 9613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:39.869148 9598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:39.869256 9598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "917900b716c54db3a390d21d31f9bad8"
format_stamp: "Formatted at 2026-06-20 23:05:39 on dist-test-slave-qgtk"
server_key: "ec80684296c9afc65608f4948770f5a4"
server_key_iv: "f381835599356fa8b7f9ebd445ba36b8"
server_key_version: "encryptionkey@0"
I20260620 23:05:39.869333 9598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:39.880657 9598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:39.883315 9598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:39.883505 9598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:39.887812 9598 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43511
I20260620 23:05:39.888034 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43511 every 8 connection(s)
I20260620 23:05:39.888298 9598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:39.889227 9666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:39.891969 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8: Bootstrap starting.
I20260620 23:05:39.892599 9666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:39.892872 9666 log.cc:824] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:39.893702 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8: No bootstrap required, opened a new log
I20260620 23:05:39.895529 9666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917900b716c54db3a390d21d31f9bad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260620 23:05:39.895687 9666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:39.895713 9666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917900b716c54db3a390d21d31f9bad8, State: Initialized, Role: FOLLOWER
I20260620 23:05:39.895879 9666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917900b716c54db3a390d21d31f9bad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260620 23:05:39.895970 9666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:39.895996 9666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:39.896027 9666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:39.896572 9666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917900b716c54db3a390d21d31f9bad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260620 23:05:39.896662 9666 leader_election.cc:302] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 917900b716c54db3a390d21d31f9bad8; no voters:
I20260620 23:05:39.896812 9666 leader_election.cc:288] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:39.896936 9669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:39.897140 9669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [term 1 LEADER]: Becoming Leader. State: Replica: 917900b716c54db3a390d21d31f9bad8, State: Running, Role: LEADER
I20260620 23:05:39.897102 9666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:39.897251 9669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917900b716c54db3a390d21d31f9bad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260620 23:05:39.897471 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9598
I20260620 23:05:39.897634 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:39.897893 21769 external_mini_cluster.cc:1428] Setting key c6aa4268bce385ec7c22debead5adf8e
I20260620 23:05:39.897975 9671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 917900b716c54db3a390d21d31f9bad8. Latest consensus state: current_term: 1 leader_uuid: "917900b716c54db3a390d21d31f9bad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917900b716c54db3a390d21d31f9bad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } } }
I20260620 23:05:39.898047 9671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:39.897866 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "917900b716c54db3a390d21d31f9bad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917900b716c54db3a390d21d31f9bad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } } }
I20260620 23:05:39.898270 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:39.898558 9678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:39.899099 9678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:39.901955 9678 catalog_manager.cc:1347] Generated new cluster ID: 54d21eb46eab4597849a2c0797c7960a
I20260620 23:05:39.902015 9678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:39.904726 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:39.898669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45729 (local address 127.0.0.1:43511)
0620 23:05:39.898838 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:39.898850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:39.898884 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:05:39.899005 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:39.899014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:39.899071 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:39.899212 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:39.899557 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:39.901104 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:39.902292 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:39.903006 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:39.903144 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:39.903910 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:39.903915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:39.903915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:39.904167 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:39.904213 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:39.904230 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:39.904323 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:39.904433 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 23:05:39.904468 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260620 23:05:39.905812 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:39.911339 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:39.907403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38905 (local address 127.0.0.1:43511)
0620 23:05:39.907464 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:39.907466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:39.907583 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:05:39.907667 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:39.907667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:39.907677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:39.907735 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:39.907898 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:39.908646 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:39.909812 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:39.910167 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:39.910224 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:39.911072 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:39.911074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:39.911074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:39.911079 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:39.911094 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:39.911098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:39.911128 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:39.911240 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:05:39.911254 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 23:05:39.916128 9678 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:39.916855 9678 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:39.924983 9678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 917900b716c54db3a390d21d31f9bad8: Generated new TSK 0
I20260620 23:05:39.925261 9678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:39.927963 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9598
W20260620 23:05:39.932240 9583 connection.cc:535] client connection to 127.0.0.1:43511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43511): Connection reset by peer (error 104)
2026-06-20T23:05:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:39 dist-test-slave-qgtk krb5kdc[9697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:39 dist-test-slave-qgtk krb5kdc[9697](info): set up 2 sockets
Jun 20 23:05:39 dist-test-slave-qgtk krb5kdc[9697](info): commencing operation
krb5kdc: starting...
W20260620 23:05:40.994308 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:41 dist-test-slave-qgtk krb5kdc[9697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:41.121124 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:41.206662 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:41.206935 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:41.206979 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:41.208189 9710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:41.208271 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:41.208293 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:41.208313 9710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:41.208344 9710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:41.209882 9710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:41.210204 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:41.210457 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:41.215653 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:41.215669 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:41.215653 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:41.216008 9710 server_base.cc:1034] running on GCE node
I20260620 23:05:41.216250 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:41.216507 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:41.217661 9710 hybrid_clock.cc:648] HybridClock initialized: now 1781996741217599 us; error 82 us; skew 500 ppm
Jun 20 23:05:41 dist-test-slave-qgtk krb5kdc[9697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:41.220463 9710 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:41.221001 9710 webserver.cc:460] Webserver started at http://127.0.0.1:41959/ using document root <none> and password file <none>
I20260620 23:05:41.221210 9710 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:41.221251 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:41.221339 9710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:41.222442 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "40a4355cb40941cea2bbbe783ce22ce2"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-qgtk"
server_key: "e4a3061d98043b168eca68cb1d982387"
server_key_iv: "fd7ea08cf9a52a9c096ed8f05a986879"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.222755 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "40a4355cb40941cea2bbbe783ce22ce2"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-qgtk"
server_key: "e4a3061d98043b168eca68cb1d982387"
server_key_iv: "fd7ea08cf9a52a9c096ed8f05a986879"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.224815 9710 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 23:05:41.227105 9725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:41.227365 9710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 23:05:41.227486 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "40a4355cb40941cea2bbbe783ce22ce2"
format_stamp: "Formatted at 2026-06-20 23:05:41 on dist-test-slave-qgtk"
server_key: "e4a3061d98043b168eca68cb1d982387"
server_key_iv: "fd7ea08cf9a52a9c096ed8f05a986879"
server_key_version: "encryptionkey@0"
I20260620 23:05:41.227576 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:41.232702 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:41.235272 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:41.235440 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:41.239722 9710 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38075
I20260620 23:05:41.239889 9777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38075 every 8 connection(s)
I20260620 23:05:41.240149 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:41.241048 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:41.243466 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2: Bootstrap starting.
I20260620 23:05:41.244066 9778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:41.244366 9778 log.cc:824] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:41.245180 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2: No bootstrap required, opened a new log
I20260620 23:05:41.246614 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9710
I20260620 23:05:41.246757 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:41.246959 21769 external_mini_cluster.cc:1428] Setting key ce892c37b22e113ca4e042e137b209ad
I20260620 23:05:41.246997 9778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a4355cb40941cea2bbbe783ce22ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38075 } }
I20260620 23:05:41.247215 9778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:41.247257 9778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a4355cb40941cea2bbbe783ce22ce2, State: Initialized, Role: FOLLOWER
I20260620 23:05:41.247428 9778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a4355cb40941cea2bbbe783ce22ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38075 } }
I20260620 23:05:41.247524 9778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:41.247586 9778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:41.247640 9778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:41.248733 9778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a4355cb40941cea2bbbe783ce22ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38075 } }
I20260620 23:05:41.248881 9778 leader_election.cc:302] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a4355cb40941cea2bbbe783ce22ce2; no voters:
I20260620 23:05:41.249043 9778 leader_election.cc:288] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:41.249091 9783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:41.249339 9783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [term 1 LEADER]: Becoming Leader. State: Replica: 40a4355cb40941cea2bbbe783ce22ce2, State: Running, Role: LEADER
I20260620 23:05:41.249408 9778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:41.249378 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.247504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51407 (local address 127.0.0.1:38075)
0620 23:05:41.247721 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.247732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.247765 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:05:41.247890 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.247897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.247926 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.248084 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.248829 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.248833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.248833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:41.248838 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.248885 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:41.248901 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.248973 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.249111 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.249160 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260620 23:05:41.249536 9783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a4355cb40941cea2bbbe783ce22ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38075 } }
I20260620 23:05:41.250025 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40a4355cb40941cea2bbbe783ce22ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a4355cb40941cea2bbbe783ce22ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38075 } } }
I20260620 23:05:41.250185 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:41.250556 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:41.250476 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a4355cb40941cea2bbbe783ce22ce2. Latest consensus state: current_term: 1 leader_uuid: "40a4355cb40941cea2bbbe783ce22ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a4355cb40941cea2bbbe783ce22ce2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38075 } } }
I20260620 23:05:41.252017 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2 [sys.catalog]: This master's current role is: LEADER
W20260620 23:05:41.251295 9799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:41.252135 9799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:41.252256 9792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:41.252468 9792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:41.254266 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:41.253103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36355 (local address 127.0.0.1:38075)
0620 23:05:41.253169 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:41.253171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:41.253180 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:05:41.253200 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:41.253201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:41.253206 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:41.253268 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:41.253981 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:41.253983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:41.253983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:41.253987 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:41.254001 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:41.254005 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:41.254056 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:41.254140 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 23:05:41.254155 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 23:05:41.254364 9792 catalog_manager.cc:1347] Generated new cluster ID: 41b85e5de55a44859bf51ebfda9f541f
I20260620 23:05:41.254401 9792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:41.274935 9792 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:41.275156 9792 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:41.286484 9792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40a4355cb40941cea2bbbe783ce22ce2: Generated new TSK 0
I20260620 23:05:41.286759 9792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:41.298370 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9710
2026-06-20T23:05:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:41 dist-test-slave-qgtk krb5kdc[9809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:41 dist-test-slave-qgtk krb5kdc[9809](info): set up 2 sockets
Jun 20 23:05:41 dist-test-slave-qgtk krb5kdc[9809](info): commencing operation
krb5kdc: starting...
W20260620 23:05:42.390450 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:42 dist-test-slave-qgtk krb5kdc[9809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:42.516566 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:42.597975 9822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:42.598214 9822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:42.598240 9822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:42.599349 9822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:42.599385 9822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:42.599399 9822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:42.599412 9822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:42.599424 9822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:42.600814 9822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:42.601053 9822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:42.601265 9822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:05:42.606312 9822 server_base.cc:1034] running on GCE node
W20260620 23:05:42.606316 9827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:42.606274 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:42.606257 9828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:42.606809 9822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:42.607107 9822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:42.608266 9822 hybrid_clock.cc:648] HybridClock initialized: now 1781996742608245 us; error 38 us; skew 500 ppm
Jun 20 23:05:42 dist-test-slave-qgtk krb5kdc[9809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:42.610808 9822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:42.611366 9822 webserver.cc:460] Webserver started at http://127.0.0.1:37373/ using document root <none> and password file <none>
I20260620 23:05:42.611596 9822 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:42.611665 9822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:42.611799 9822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:42.612859 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "9c64ad6537a04702aa6dfc715a5b7c82"
format_stamp: "Formatted at 2026-06-20 23:05:42 on dist-test-slave-qgtk"
server_key: "cb9f7138639a17be3f05a6f3520518cd"
server_key_iv: "43a5c705925515252b073346c069a37a"
server_key_version: "encryptionkey@0"
I20260620 23:05:42.613229 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "9c64ad6537a04702aa6dfc715a5b7c82"
format_stamp: "Formatted at 2026-06-20 23:05:42 on dist-test-slave-qgtk"
server_key: "cb9f7138639a17be3f05a6f3520518cd"
server_key_iv: "43a5c705925515252b073346c069a37a"
server_key_version: "encryptionkey@0"
I20260620 23:05:42.614651 9822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:42.615411 9837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:42.615590 9822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:05:42.615688 9822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "9c64ad6537a04702aa6dfc715a5b7c82"
format_stamp: "Formatted at 2026-06-20 23:05:42 on dist-test-slave-qgtk"
server_key: "cb9f7138639a17be3f05a6f3520518cd"
server_key_iv: "43a5c705925515252b073346c069a37a"
server_key_version: "encryptionkey@0"
I20260620 23:05:42.615773 9822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:42.632568 9822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:42.635102 9822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:42.635289 9822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:42.639578 9822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38153
I20260620 23:05:42.639606 9889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38153 every 8 connection(s)
I20260620 23:05:42.639982 9822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:42.640831 9890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:42.641703 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9822
I20260620 23:05:42.641793 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:42.642009 21769 external_mini_cluster.cc:1428] Setting key e1b55b1249b03d94152f8cd9782f32e7
I20260620 23:05:42.644001 9890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82: Bootstrap starting.
I20260620 23:05:42.644681 9890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:42.645002 9890 log.cc:824] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:42.645745 9890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82: No bootstrap required, opened a new log
I20260620 23:05:42.648305 9890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260620 23:05:42.648543 9890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:42.648626 9890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c64ad6537a04702aa6dfc715a5b7c82, State: Initialized, Role: FOLLOWER
Jun 20 23:05:42 dist-test-slave-qgtk krb5kdc[9809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:05:42.648829 9890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260620 23:05:42.648952 9890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:42.649019 9890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:42.649089 9890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:42.649977 9890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260620 23:05:42.650152 9890 leader_election.cc:302] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c64ad6537a04702aa6dfc715a5b7c82; no voters:
I20260620 23:05:42.650348 9890 leader_election.cc:288] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:42.650694 9890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:42.651203 9895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:42.651368 9895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [term 1 LEADER]: Becoming Leader. State: Replica: 9c64ad6537a04702aa6dfc715a5b7c82, State: Running, Role: LEADER
I20260620 23:05:42.651530 9895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } }
I20260620 23:05:42.651948 9900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } } }
I20260620 23:05:42.652042 9900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:42.652196 9900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c64ad6537a04702aa6dfc715a5b7c82. Latest consensus state: current_term: 1 leader_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c64ad6537a04702aa6dfc715a5b7c82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38153 } } }
I20260620 23:05:42.652251 9900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82 [sys.catalog]: This master's current role is: LEADER
W20260620 23:05:42.652611 9909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:42.652694 9909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:42.652801 9910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:42.653030 9910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:42.654001 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:42.642594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32809 (local address 127.0.0.1:38153)
0620 23:05:42.642815 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:42.642823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:42.642847 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:05:42.643268 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:42.643275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:42.643321 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:42.643468 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:42.644099 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:42.645088 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:42.645880 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:42.646357 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:42.646484 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:42.649301 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:42.649306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:42.649306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:42.649320 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:42.652151 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:42.652590 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:42.652593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:42.652594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:42.652698 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:42.652911 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:42.652912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:42.652913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:42.653241 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:42.653387 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:42.653732 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 23:05:42.653853 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260620 23:05:42.654886 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:42.655176 9910 catalog_manager.cc:1347] Generated new cluster ID: a125dd57d03c408ab01e62823381e461
I20260620 23:05:42.655220 9910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:42.663178 9910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:42.663817 9910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:42.664076 9893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:42.656419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38593 (local address 127.0.0.1:38153)
0620 23:05:42.656460 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:42.656462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:42.656497 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:05:42.656519 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:42.656520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:42.656530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:42.656609 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:42.657195 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:42.658033 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:42.659111 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:42.659490 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:42.659575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:42.660932 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:42.660934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:42.660934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:42.660942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:42.662757 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:42.663201 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:42.663202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:42.663203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:42.663271 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:42.663486 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:42.663487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:42.663488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:42.663641 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:42.663753 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:42.663867 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:05:42.663963 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:05:42.667153 9910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c64ad6537a04702aa6dfc715a5b7c82: Generated new TSK 0
I20260620 23:05:42.667371 9910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:42.673612 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9822
2026-06-20T23:05:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:42 dist-test-slave-qgtk krb5kdc[9921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:42 dist-test-slave-qgtk krb5kdc[9921](info): set up 2 sockets
Jun 20 23:05:42 dist-test-slave-qgtk krb5kdc[9921](info): commencing operation
krb5kdc: starting...
W20260620 23:05:43.734409 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:43 dist-test-slave-qgtk krb5kdc[9921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:43.862325 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:43.951946 9934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:43.952168 9934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:43.952195 9934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:43.953358 9934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:43.953397 9934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:43.953413 9934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:43.953429 9934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:43.953441 9934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:43.954962 9934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:43.955169 9934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:43.955356 9934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:43.960426 9940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:43.960556 9939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:43.960426 9942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:43.960794 9934 server_base.cc:1034] running on GCE node
I20260620 23:05:43.961043 9934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:43.961409 9934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:43.962601 9934 hybrid_clock.cc:648] HybridClock initialized: now 1781996743962572 us; error 46 us; skew 500 ppm
Jun 20 23:05:43 dist-test-slave-qgtk krb5kdc[9921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:43.965322 9934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:43.965859 9934 webserver.cc:460] Webserver started at http://127.0.0.1:44099/ using document root <none> and password file <none>
I20260620 23:05:43.966105 9934 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:43.966235 9934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:43.966403 9934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:43.967656 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "cb11f90aff9b4be0ae22059ef99a6bc5"
format_stamp: "Formatted at 2026-06-20 23:05:43 on dist-test-slave-qgtk"
server_key: "ab51d44f0c083343048a526f2085d3f7"
server_key_iv: "fd4b2579fb90a10be123efb05591e8ca"
server_key_version: "encryptionkey@0"
I20260620 23:05:43.968039 9934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "cb11f90aff9b4be0ae22059ef99a6bc5"
format_stamp: "Formatted at 2026-06-20 23:05:43 on dist-test-slave-qgtk"
server_key: "ab51d44f0c083343048a526f2085d3f7"
server_key_iv: "fd4b2579fb90a10be123efb05591e8ca"
server_key_version: "encryptionkey@0"
I20260620 23:05:43.969592 9934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:05:43.970739 9949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:43.971016 9934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:05:43.971134 9934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "cb11f90aff9b4be0ae22059ef99a6bc5"
format_stamp: "Formatted at 2026-06-20 23:05:43 on dist-test-slave-qgtk"
server_key: "ab51d44f0c083343048a526f2085d3f7"
server_key_iv: "fd4b2579fb90a10be123efb05591e8ca"
server_key_version: "encryptionkey@0"
I20260620 23:05:43.971218 9934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:44.001408 9934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:44.004016 9934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:44.004220 9934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:44.008926 9934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40493
I20260620 23:05:44.008991 10001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40493 every 8 connection(s)
I20260620 23:05:44.009394 9934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:44.010358 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:44.012701 10002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5: Bootstrap starting.
I20260620 23:05:44.013252 10002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:44.013499 10002 log.cc:824] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:44.014451 10002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5: No bootstrap required, opened a new log
I20260620 23:05:44.016233 10002 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260620 23:05:44.016398 10002 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:44.016425 10002 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb11f90aff9b4be0ae22059ef99a6bc5, State: Initialized, Role: FOLLOWER
I20260620 23:05:44.016582 10002 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260620 23:05:44.016671 10002 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:44.016696 10002 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:44.016729 10002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:44.017261 10002 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260620 23:05:44.017362 10002 leader_election.cc:302] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb11f90aff9b4be0ae22059ef99a6bc5; no voters:
I20260620 23:05:44.017529 10002 leader_election.cc:288] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:44.017560 10005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:44.017688 10005 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [term 1 LEADER]: Becoming Leader. State: Replica: cb11f90aff9b4be0ae22059ef99a6bc5, State: Running, Role: LEADER
I20260620 23:05:44.017814 10002 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:44.017849 10005 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } }
I20260620 23:05:44.018086 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 9934
I20260620 23:05:44.018242 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:44.018484 21769 external_mini_cluster.cc:1428] Setting key 817bfe65262219692ea078450aaff9dd
I20260620 23:05:44.018533 10006 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } } }
I20260620 23:05:44.018636 10006 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:44.018668 10007 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb11f90aff9b4be0ae22059ef99a6bc5. Latest consensus state: current_term: 1 leader_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb11f90aff9b4be0ae22059ef99a6bc5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40493 } } }
I20260620 23:05:44.018762 10007 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:44.018926 10013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:44.019500 10013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:44.021219 10013 catalog_manager.cc:1347] Generated new cluster ID: 1063bebe459f41229a9d65458e787638
I20260620 23:05:44.021279 10013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:44.025504 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:44.019215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45491 (local address 127.0.0.1:40493)
0620 23:05:44.019940 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:44.019950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:44.019991 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:05:44.020163 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:44.020172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:44.020243 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:44.020416 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:44.020631 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.021978 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.023235 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.023870 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.023999 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:44.024760 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:44.024765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:44.024765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:44.024999 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:44.025040 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:44.025055 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:44.025137 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:44.025206 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:05:44.025283 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":130,"threads_started":1}
I20260620 23:05:44.026628 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:44.031903 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:44.028226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36097 (local address 127.0.0.1:40493)
0620 23:05:44.028267 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:44.028269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:44.028498 (+ 229us) server_negotiation.cc:407] Connection header received
0620 23:05:44.028532 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:44.028532 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:44.028545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:44.028613 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:44.028734 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.029422 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.030542 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:44.030835 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:44.030920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:44.031626 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:44.031628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:44.031628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:44.031633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:44.031652 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:44.031657 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:44.031726 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:44.031791 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 23:05:44.031819 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:05:44.065012 10013 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:44.065640 10013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:44.068276 10013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb11f90aff9b4be0ae22059ef99a6bc5: Generated new TSK 0
I20260620 23:05:44.068528 10013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:44.079747 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 9934
2026-06-20T23:05:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:44 dist-test-slave-qgtk krb5kdc[10033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:44 dist-test-slave-qgtk krb5kdc[10033](info): set up 2 sockets
Jun 20 23:05:44 dist-test-slave-qgtk krb5kdc[10033](info): commencing operation
krb5kdc: starting...
W20260620 23:05:45.234428 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:45 dist-test-slave-qgtk krb5kdc[10033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:45.361012 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:45.444473 10046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:45.444748 10046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:45.444792 10046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:45.446017 10046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:45.446095 10046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:45.446117 10046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:45.446169 10046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:45.446203 10046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:45.447777 10046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:45.448074 10046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:45.448328 10046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:45.453559 10051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:45.453507 10052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:45.453511 10054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:45.454010 10046 server_base.cc:1034] running on GCE node
I20260620 23:05:45.454314 10046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:45.454656 10046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:45.455834 10046 hybrid_clock.cc:648] HybridClock initialized: now 1781996745455820 us; error 28 us; skew 500 ppm
Jun 20 23:05:45 dist-test-slave-qgtk krb5kdc[10033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:45.463050 10046 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:45.463572 10046 webserver.cc:460] Webserver started at http://127.0.0.1:39243/ using document root <none> and password file <none>
I20260620 23:05:45.463840 10046 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:45.463917 10046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:45.464069 10046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:45.465169 10046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "df87c07d144a440499b6bd8d677da57f"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-qgtk"
server_key: "2c0c801ec0ceef42604a5a4a443adf27"
server_key_iv: "a9c30311be251c67b76796f8e6bfa570"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.465538 10046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "df87c07d144a440499b6bd8d677da57f"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-qgtk"
server_key: "2c0c801ec0ceef42604a5a4a443adf27"
server_key_iv: "a9c30311be251c67b76796f8e6bfa570"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.467130 10046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:05:45.468037 10061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:45.468225 10046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:45.468338 10046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "df87c07d144a440499b6bd8d677da57f"
format_stamp: "Formatted at 2026-06-20 23:05:45 on dist-test-slave-qgtk"
server_key: "2c0c801ec0ceef42604a5a4a443adf27"
server_key_iv: "a9c30311be251c67b76796f8e6bfa570"
server_key_version: "encryptionkey@0"
I20260620 23:05:45.468417 10046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:45.471994 10046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:45.474826 10046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:45.475008 10046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:45.479657 10046 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38015
I20260620 23:05:45.479696 10113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38015 every 8 connection(s)
I20260620 23:05:45.480060 10046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:45.481067 10114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:45.483585 10114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f: Bootstrap starting.
I20260620 23:05:45.484134 10114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:45.484380 10114 log.cc:824] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:45.485105 10114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f: No bootstrap required, opened a new log
I20260620 23:05:45.486341 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10046
I20260620 23:05:45.486495 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:45.486699 21769 external_mini_cluster.cc:1428] Setting key 0626aa34eae4c5684a6070606e10f50d
I20260620 23:05:45.486943 10114 raft_consensus.cc:348] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df87c07d144a440499b6bd8d677da57f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38015 } }
I20260620 23:05:45.487133 10114 raft_consensus.cc:374] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:45.487178 10114 raft_consensus.cc:729] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df87c07d144a440499b6bd8d677da57f, State: Initialized, Role: FOLLOWER
I20260620 23:05:45.487340 10114 consensus_queue.cc:260] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df87c07d144a440499b6bd8d677da57f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38015 } }
I20260620 23:05:45.487437 10114 raft_consensus.cc:388] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:45.487473 10114 raft_consensus.cc:482] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:45.487522 10114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:45.488641 10114 raft_consensus.cc:504] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df87c07d144a440499b6bd8d677da57f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38015 } }
I20260620 23:05:45.488775 10114 leader_election.cc:302] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df87c07d144a440499b6bd8d677da57f; no voters:
I20260620 23:05:45.488943 10114 leader_election.cc:288] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:45.489001 10119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:45.489192 10114 sys_catalog.cc:564] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:45.489260 10119 raft_consensus.cc:686] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [term 1 LEADER]: Becoming Leader. State: Replica: df87c07d144a440499b6bd8d677da57f, State: Running, Role: LEADER
I20260620 23:05:45.489221 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.487250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41815 (local address 127.0.0.1:38015)
0620 23:05:45.487432 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.487442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.487478 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:05:45.487610 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.487617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.487649 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.487810 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.488447 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.488451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.488451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:45.488455 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.488502 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:45.488517 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.488595 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.488733 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.488898 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260620 23:05:45.489403 10119 consensus_queue.cc:237] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df87c07d144a440499b6bd8d677da57f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38015 } }
W20260620 23:05:45.491114 10133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:45.491173 10133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:45.491200 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:45.490635 10121 sys_catalog.cc:455] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader df87c07d144a440499b6bd8d677da57f. Latest consensus state: current_term: 1 leader_uuid: "df87c07d144a440499b6bd8d677da57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df87c07d144a440499b6bd8d677da57f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38015 } } }
I20260620 23:05:45.492746 10121 sys_catalog.cc:458] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:45.490799 10120 sys_catalog.cc:455] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df87c07d144a440499b6bd8d677da57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df87c07d144a440499b6bd8d677da57f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38015 } } }
I20260620 23:05:45.492986 10140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:45.493011 10120 sys_catalog.cc:458] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:45.493172 10140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:45.494685 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:45.493032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54541 (local address 127.0.0.1:38015)
0620 23:05:45.493152 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:45.493154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:45.493164 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:05:45.493217 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:45.493218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:45.493224 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:45.493274 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:45.494424 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:45.494425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:45.494426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:45.494430 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:45.494448 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:45.494453 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:45.494523 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:45.494592 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 23:05:45.494606 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:05:45.495043 10140 catalog_manager.cc:1347] Generated new cluster ID: cf87288eded54b38a24a38f5c18db2df
I20260620 23:05:45.495105 10140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:45.525132 10140 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:45.525358 10140 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:45.532929 10140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df87c07d144a440499b6bd8d677da57f: Generated new TSK 0
I20260620 23:05:45.533190 10140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:45.537910 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10046
2026-06-20T23:05:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:45 dist-test-slave-qgtk krb5kdc[10145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:45 dist-test-slave-qgtk krb5kdc[10145](info): set up 2 sockets
Jun 20 23:05:45 dist-test-slave-qgtk krb5kdc[10145](info): commencing operation
krb5kdc: starting...
W20260620 23:05:46.626271 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:46 dist-test-slave-qgtk krb5kdc[10145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:46.749405 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:46.833957 10158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:46.834252 10158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:46.834296 10158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:46.835613 10158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:46.835680 10158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:46.835704 10158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:46.835726 10158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:46.835754 10158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:46.837399 10158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:46.837687 10158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:46.837945 10158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:46.842906 10164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:46.842897 10166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:46.842904 10163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:46.843329 10158 server_base.cc:1034] running on GCE node
I20260620 23:05:46.843592 10158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:46.843932 10158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:46.845155 10158 hybrid_clock.cc:648] HybridClock initialized: now 1781996746845131 us; error 30 us; skew 500 ppm
Jun 20 23:05:46 dist-test-slave-qgtk krb5kdc[10145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:46.847738 10158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:46.848242 10158 webserver.cc:460] Webserver started at http://127.0.0.1:42229/ using document root <none> and password file <none>
I20260620 23:05:46.848469 10158 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:46.848532 10158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:46.848678 10158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:46.849757 10158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "dbd99ecbf1634bb0a3cf137002984b2e"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-qgtk"
server_key: "dfeb813f49b4ad00a2cb3cd28b0786e5"
server_key_iv: "1795038243c79014b574b05796af9a02"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.850178 10158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "dbd99ecbf1634bb0a3cf137002984b2e"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-qgtk"
server_key: "dfeb813f49b4ad00a2cb3cd28b0786e5"
server_key_iv: "1795038243c79014b574b05796af9a02"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.851509 10158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:46.852283 10173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:46.852444 10158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 23:05:46.852540 10158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "dbd99ecbf1634bb0a3cf137002984b2e"
format_stamp: "Formatted at 2026-06-20 23:05:46 on dist-test-slave-qgtk"
server_key: "dfeb813f49b4ad00a2cb3cd28b0786e5"
server_key_iv: "1795038243c79014b574b05796af9a02"
server_key_version: "encryptionkey@0"
I20260620 23:05:46.852627 10158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:46.866540 10158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:46.868489 10158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:46.868656 10158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:46.872602 10158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37295
I20260620 23:05:46.872645 10225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37295 every 8 connection(s)
I20260620 23:05:46.873016 10158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:46.873879 10226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:46.874416 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10158
I20260620 23:05:46.874543 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:46.874728 21769 external_mini_cluster.cc:1428] Setting key f5c1ab15639e872a88e116f8a12daccf
I20260620 23:05:46.876663 10226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e: Bootstrap starting.
I20260620 23:05:46.877259 10226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:46.877563 10226 log.cc:824] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:46.878351 10226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e: No bootstrap required, opened a new log
I20260620 23:05:46.880347 10226 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 23:05:46.880544 10226 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:46.880622 10226 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd99ecbf1634bb0a3cf137002984b2e, State: Initialized, Role: FOLLOWER
I20260620 23:05:46.880774 10226 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 23:05:46.880865 10226 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:46.880910 10226 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:46.880959 10226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:46.881587 10226 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 23:05:46.881719 10226 leader_election.cc:302] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbd99ecbf1634bb0a3cf137002984b2e; no voters:
I20260620 23:05:46.881888 10226 leader_election.cc:288] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:46.882037 10231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:46.882192 10226 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 23:05:46 dist-test-slave-qgtk krb5kdc[10145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:05:46.882280 10231 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [term 1 LEADER]: Becoming Leader. State: Replica: dbd99ecbf1634bb0a3cf137002984b2e, State: Running, Role: LEADER
I20260620 23:05:46.882372 10231 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } }
I20260620 23:05:46.882732 10232 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } } }
I20260620 23:05:46.882725 10231 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd99ecbf1634bb0a3cf137002984b2e. Latest consensus state: current_term: 1 leader_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd99ecbf1634bb0a3cf137002984b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37295 } } }
I20260620 23:05:46.882826 10232 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:46.882834 10231 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:46.883733 10240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:46.884478 10240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:46.886175 10240 catalog_manager.cc:1347] Generated new cluster ID: ecce05d1864b44758965322f20a490e4
I20260620 23:05:46.886238 10240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:46.887205 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.875335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:58617 (local address 127.0.0.1:37295)
0620 23:05:46.875526 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.875536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.875571 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:05:46.876013 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.876020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.876086 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.876244 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.877038 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.878299 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.879143 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.879610 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.879755 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.882913 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.882918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.882919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.882933 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.885379 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.885908 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.885912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.885912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.886054 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.886340 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.886341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.886342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.886597 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.886786 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.886885 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.886993 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260620 23:05:46.888191 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:46.896291 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:46.889511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:53241 (local address 127.0.0.1:37295)
0620 23:05:46.889551 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:46.889553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:46.890274 (+ 721us) server_negotiation.cc:407] Connection header received
0620 23:05:46.890349 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:46.890350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:46.890361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:46.890415 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:46.890950 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.891448 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.892201 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:46.892440 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:46.892514 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:46.893709 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:46.893710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:46.893710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:46.893718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:46.895132 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.895491 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.895492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.895493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.895557 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:46.895770 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:46.895771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:46.895771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:46.895881 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:46.895992 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:46.896114 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:05:46.896176 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":20}
I20260620 23:05:46.912356 10240 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:46.913178 10240 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:46.927791 10240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd99ecbf1634bb0a3cf137002984b2e: Generated new TSK 0
I20260620 23:05:46.928066 10240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:46.932469 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10158
W20260620 23:05:46.936659 10143 connection.cc:535] client connection to 127.0.0.1:37295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37295): Connection reset by peer (error 104)
2026-06-20T23:05:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:46 dist-test-slave-qgtk krb5kdc[10257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:46 dist-test-slave-qgtk krb5kdc[10257](info): set up 2 sockets
Jun 20 23:05:46 dist-test-slave-qgtk krb5kdc[10257](info): commencing operation
krb5kdc: starting...
W20260620 23:05:48.079025 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:48 dist-test-slave-qgtk krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:48.203141 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:48.284798 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:48.285075 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:48.285115 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:48.286336 10270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:48.286410 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:48.286435 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:48.286458 10270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:48.286484 10270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:48.287992 10270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:48.288282 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:48.288548 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:48.293654 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:48.293612 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:48.293628 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:48.293659 10270 server_base.cc:1034] running on GCE node
I20260620 23:05:48.294188 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:48.294589 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:48.295769 10270 hybrid_clock.cc:648] HybridClock initialized: now 1781996748295742 us; error 38 us; skew 500 ppm
Jun 20 23:05:48 dist-test-slave-qgtk krb5kdc[10257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:48.298343 10270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:48.298738 10270 webserver.cc:460] Webserver started at http://127.0.0.1:39943/ using document root <none> and password file <none>
I20260620 23:05:48.298923 10270 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:48.298960 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:48.299062 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:48.300060 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "f4e968b4c8f747d7aca2b7530aa52fe9"
format_stamp: "Formatted at 2026-06-20 23:05:48 on dist-test-slave-qgtk"
server_key: "ba3020c5d00196abc14d264c8467d9e2"
server_key_iv: "c1598cdcedb35f51a9866c5ecae77fd5"
server_key_version: "encryptionkey@0"
I20260620 23:05:48.300364 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "f4e968b4c8f747d7aca2b7530aa52fe9"
format_stamp: "Formatted at 2026-06-20 23:05:48 on dist-test-slave-qgtk"
server_key: "ba3020c5d00196abc14d264c8467d9e2"
server_key_iv: "c1598cdcedb35f51a9866c5ecae77fd5"
server_key_version: "encryptionkey@0"
I20260620 23:05:48.301676 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 23:05:48.302491 10285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:48.302641 10270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:05:48.302729 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "f4e968b4c8f747d7aca2b7530aa52fe9"
format_stamp: "Formatted at 2026-06-20 23:05:48 on dist-test-slave-qgtk"
server_key: "ba3020c5d00196abc14d264c8467d9e2"
server_key_iv: "c1598cdcedb35f51a9866c5ecae77fd5"
server_key_version: "encryptionkey@0"
I20260620 23:05:48.302791 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:48.312085 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:48.314452 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:48.314569 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:48.318531 10270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33557
I20260620 23:05:48.318563 10337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33557 every 8 connection(s)
I20260620 23:05:48.318921 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:48.319916 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:48.322587 10338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9: Bootstrap starting.
I20260620 23:05:48.323237 10338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:48.323534 10338 log.cc:824] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:48.324275 10338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9: No bootstrap required, opened a new log
I20260620 23:05:48.325982 10338 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33557 } }
I20260620 23:05:48.326193 10338 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:48.326243 10338 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e968b4c8f747d7aca2b7530aa52fe9, State: Initialized, Role: FOLLOWER
I20260620 23:05:48.326382 10338 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33557 } }
I20260620 23:05:48.326469 10338 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:48.326493 10338 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:48.326570 10338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:48.327113 10338 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33557 } }
I20260620 23:05:48.327214 10338 leader_election.cc:302] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4e968b4c8f747d7aca2b7530aa52fe9; no voters:
I20260620 23:05:48.327369 10338 leader_election.cc:288] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:48.327417 10341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:48.327562 10341 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [term 1 LEADER]: Becoming Leader. State: Replica: f4e968b4c8f747d7aca2b7530aa52fe9, State: Running, Role: LEADER
I20260620 23:05:48.327654 10338 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:48.327787 10341 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33557 } }
I20260620 23:05:48.328070 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10270
I20260620 23:05:48.328194 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:48.328266 10342 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33557 } } }
I20260620 23:05:48.328356 10342 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:48.328402 21769 external_mini_cluster.cc:1428] Setting key 901a0aeffa2bbc81eb670c66ae4df3c8
I20260620 23:05:48.328680 10343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4e968b4c8f747d7aca2b7530aa52fe9. Latest consensus state: current_term: 1 leader_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4e968b4c8f747d7aca2b7530aa52fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33557 } } }
I20260620 23:05:48.328756 10343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:48.328943 10345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:48.329798 10345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:48.331570 10345 catalog_manager.cc:1347] Generated new cluster ID: bad1e9fe6b954f5baf360479f676f5b8
I20260620 23:05:48.331622 10345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:48.334826 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.329159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:32769 (local address 127.0.0.1:33557)
0620 23:05:48.329361 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.329371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.329403 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:05:48.329526 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.329535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.329618 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.329802 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.330070 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.331315 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.332554 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.333170 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.333266 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.334060 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.334064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.334064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:48.334294 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.334340 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:48.334355 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.334438 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.334582 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.334612 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260620 23:05:48.335901 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:48.341403 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:48.337574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:39425 (local address 127.0.0.1:33557)
0620 23:05:48.337618 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:48.337620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:48.337627 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:05:48.337717 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:48.337718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:48.337730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:48.337797 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:48.337912 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.338806 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.339841 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:48.340196 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:48.340280 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:48.341111 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:48.341113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:48.341113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:48.341119 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:48.341138 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:48.341144 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:48.341220 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:48.341282 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 23:05:48.341297 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 23:05:48.343575 10345 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:48.344374 10345 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:48.357683 10345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4e968b4c8f747d7aca2b7530aa52fe9: Generated new TSK 0
I20260620 23:05:48.357966 10345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:48.363152 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10270
W20260620 23:05:48.367662 10255 connection.cc:535] client connection to 127.0.0.1:33557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33557): Connection reset by peer (error 104)
2026-06-20T23:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:48 dist-test-slave-qgtk krb5kdc[10369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:48 dist-test-slave-qgtk krb5kdc[10369](info): set up 2 sockets
Jun 20 23:05:48 dist-test-slave-qgtk krb5kdc[10369](info): commencing operation
krb5kdc: starting...
W20260620 23:05:49.450443 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:49 dist-test-slave-qgtk krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:49.577183 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:49.661895 10382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:49.662170 10382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:49.662220 10382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:49.663394 10382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:49.663460 10382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:49.663496 10382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:49.663525 10382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:49.663553 10382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:49.665050 10382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:49.665323 10382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:49.665570 10382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:05:49.670715 10382 server_base.cc:1034] running on GCE node
W20260620 23:05:49.670693 10387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:49.670693 10390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:49.670655 10388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:49.671231 10382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:49.671586 10382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:49.672767 10382 hybrid_clock.cc:648] HybridClock initialized: now 1781996749672767 us; error 46 us; skew 500 ppm
Jun 20 23:05:49 dist-test-slave-qgtk krb5kdc[10369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:49.675686 10382 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:49.676151 10382 webserver.cc:460] Webserver started at http://127.0.0.1:45417/ using document root <none> and password file <none>
I20260620 23:05:49.676412 10382 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:49.676481 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:49.676628 10382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:49.677732 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "1843dccb1d1a44479afb8a9c73123bd9"
format_stamp: "Formatted at 2026-06-20 23:05:49 on dist-test-slave-qgtk"
server_key: "4e03f490c8322aa9bcc386ec9e701241"
server_key_iv: "ebc20adeacbea1dd95c1d09bc6d5ae63"
server_key_version: "encryptionkey@0"
I20260620 23:05:49.678095 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "1843dccb1d1a44479afb8a9c73123bd9"
format_stamp: "Formatted at 2026-06-20 23:05:49 on dist-test-slave-qgtk"
server_key: "4e03f490c8322aa9bcc386ec9e701241"
server_key_iv: "ebc20adeacbea1dd95c1d09bc6d5ae63"
server_key_version: "encryptionkey@0"
I20260620 23:05:49.679503 10382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:49.680600 10397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:49.680866 10382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:49.680974 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "1843dccb1d1a44479afb8a9c73123bd9"
format_stamp: "Formatted at 2026-06-20 23:05:49 on dist-test-slave-qgtk"
server_key: "4e03f490c8322aa9bcc386ec9e701241"
server_key_iv: "ebc20adeacbea1dd95c1d09bc6d5ae63"
server_key_version: "encryptionkey@0"
I20260620 23:05:49.681067 10382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:49.685119 10382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:49.687225 10382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:49.687381 10382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:49.691248 10382 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41235
I20260620 23:05:49.691293 10449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41235 every 8 connection(s)
I20260620 23:05:49.691628 10382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:49.692209 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10382
I20260620 23:05:49.692337 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:49.692528 21769 external_mini_cluster.cc:1428] Setting key 6429debae218008396e9acc6b45a386b
I20260620 23:05:49.692778 10450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:49.695247 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:49.693053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:54909 (local address 127.0.0.1:41235)
0620 23:05:49.693256 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:49.693266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:49.693297 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:05:49.693876 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:49.693884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:49.693918 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:49.694066 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:49.694734 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:49.694739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:49.694740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:49.694744 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:49.694786 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:49.694800 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:49.694878 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:49.694912 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 23:05:49.695052 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260620 23:05:49.696039 10450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9: Bootstrap starting.
I20260620 23:05:49.696244 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:49.696640 10450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:49.696938 10450 log.cc:824] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:49.697770 10450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9: No bootstrap required, opened a new log
I20260620 23:05:49.698761 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:49.697599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:51589 (local address 127.0.0.1:41235)
0620 23:05:49.697631 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:49.697633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:49.697748 (+ 115us) server_negotiation.cc:407] Connection header received
0620 23:05:49.697823 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:49.697824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:49.697829 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:49.697875 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:49.698549 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:49.698550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:49.698551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:49.698555 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:49.698568 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:49.698572 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:49.698627 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:49.698684 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 23:05:49.698698 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 23:05:49.699787 10450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1843dccb1d1a44479afb8a9c73123bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41235 } }
I20260620 23:05:49.699918 10450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:49.699939 10450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1843dccb1d1a44479afb8a9c73123bd9, State: Initialized, Role: FOLLOWER
I20260620 23:05:49.700049 10450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1843dccb1d1a44479afb8a9c73123bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41235 } }
I20260620 23:05:49.700127 10450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:49.700155 10450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:49.700193 10450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:49.700687 10450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1843dccb1d1a44479afb8a9c73123bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41235 } }
I20260620 23:05:49.700769 10450 leader_election.cc:302] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1843dccb1d1a44479afb8a9c73123bd9; no voters:
I20260620 23:05:49.700882 10450 leader_election.cc:288] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:49.700950 10460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:49.701098 10450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:49.701125 10460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [term 1 LEADER]: Becoming Leader. State: Replica: 1843dccb1d1a44479afb8a9c73123bd9, State: Running, Role: LEADER
I20260620 23:05:49.701251 10460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1843dccb1d1a44479afb8a9c73123bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41235 } }
I20260620 23:05:49.701638 10461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1843dccb1d1a44479afb8a9c73123bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1843dccb1d1a44479afb8a9c73123bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41235 } } }
I20260620 23:05:49.701640 10462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1843dccb1d1a44479afb8a9c73123bd9. Latest consensus state: current_term: 1 leader_uuid: "1843dccb1d1a44479afb8a9c73123bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1843dccb1d1a44479afb8a9c73123bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41235 } } }
I20260620 23:05:49.701891 10462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:49.701808 10461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:49.702312 10469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:49.702767 10469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:49.704437 10469 catalog_manager.cc:1347] Generated new cluster ID: 011395714b564ad28ffa54aefa31c6b6
I20260620 23:05:49.704491 10469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:49.717927 10469 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:49.718178 10469 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:49.727141 10469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1843dccb1d1a44479afb8a9c73123bd9: Generated new TSK 0
I20260620 23:05:49.727360 10469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:49.732472 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10382
2026-06-20T23:05:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:49 dist-test-slave-qgtk krb5kdc[10481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:49 dist-test-slave-qgtk krb5kdc[10481](info): set up 2 sockets
Jun 20 23:05:49 dist-test-slave-qgtk krb5kdc[10481](info): commencing operation
krb5kdc: starting...
W20260620 23:05:50.770773 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:50 dist-test-slave-qgtk krb5kdc[10481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:50.898589 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:50.982589 10494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:50.982865 10494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:50.982895 10494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:50.984076 10494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:50.984120 10494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:50.984134 10494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:50.984148 10494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:50.984205 10494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:50.985697 10494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:50.985957 10494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:50.986214 10494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:05:50.991585 10494 server_base.cc:1034] running on GCE node
W20260620 23:05:50.991523 10502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:50.991514 10499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:50.991541 10500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:50.992035 10494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:50.992358 10494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:50.993546 10494 hybrid_clock.cc:648] HybridClock initialized: now 1781996750993522 us; error 39 us; skew 500 ppm
Jun 20 23:05:50 dist-test-slave-qgtk krb5kdc[10481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:51.001180 10494 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:51.001746 10494 webserver.cc:460] Webserver started at http://127.0.0.1:45311/ using document root <none> and password file <none>
I20260620 23:05:51.002022 10494 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:51.002096 10494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:51.002278 10494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:51.003443 10494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "cff23b1581dc4db5ad5b824e8969aea2"
format_stamp: "Formatted at 2026-06-20 23:05:51 on dist-test-slave-qgtk"
server_key: "296c265efc97a05e6e9c99b99ea6fd49"
server_key_iv: "cd5db1ec5ef5f1936bcbcc7848d6df18"
server_key_version: "encryptionkey@0"
I20260620 23:05:51.003850 10494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "cff23b1581dc4db5ad5b824e8969aea2"
format_stamp: "Formatted at 2026-06-20 23:05:51 on dist-test-slave-qgtk"
server_key: "296c265efc97a05e6e9c99b99ea6fd49"
server_key_iv: "cd5db1ec5ef5f1936bcbcc7848d6df18"
server_key_version: "encryptionkey@0"
I20260620 23:05:51.005383 10494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:51.006397 10509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:51.006639 10494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:51.006757 10494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "cff23b1581dc4db5ad5b824e8969aea2"
format_stamp: "Formatted at 2026-06-20 23:05:51 on dist-test-slave-qgtk"
server_key: "296c265efc97a05e6e9c99b99ea6fd49"
server_key_iv: "cd5db1ec5ef5f1936bcbcc7848d6df18"
server_key_version: "encryptionkey@0"
I20260620 23:05:51.006848 10494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:51.022557 10494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:51.025125 10494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:51.025358 10494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:51.030033 10494 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40459
I20260620 23:05:51.030088 10561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40459 every 8 connection(s)
I20260620 23:05:51.030532 10494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:51.031549 10562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:51.033681 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10494
I20260620 23:05:51.033776 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:51.033967 21769 external_mini_cluster.cc:1428] Setting key 03460c74d6bd8a7444b6b393b48cd763
I20260620 23:05:51.034224 10562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2: Bootstrap starting.
I20260620 23:05:51.035053 10562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:51.035385 10562 log.cc:824] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:51.036113 10562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2: No bootstrap required, opened a new log
I20260620 23:05:51.038004 10562 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff23b1581dc4db5ad5b824e8969aea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260620 23:05:51.038198 10562 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:51.038245 10562 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff23b1581dc4db5ad5b824e8969aea2, State: Initialized, Role: FOLLOWER
I20260620 23:05:51.038389 10562 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff23b1581dc4db5ad5b824e8969aea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260620 23:05:51.038479 10562 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:51.038509 10562 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:51.038554 10562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:51.039301 10562 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff23b1581dc4db5ad5b824e8969aea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260620 23:05:51.039445 10562 leader_election.cc:302] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff23b1581dc4db5ad5b824e8969aea2; no voters:
I20260620 23:05:51.039620 10562 leader_election.cc:288] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:51.039633 10567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:51.039855 10567 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [term 1 LEADER]: Becoming Leader. State: Replica: cff23b1581dc4db5ad5b824e8969aea2, State: Running, Role: LEADER
I20260620 23:05:51.039929 10562 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:51.039956 10567 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff23b1581dc4db5ad5b824e8969aea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } }
I20260620 23:05:51.040287 10567 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff23b1581dc4db5ad5b824e8969aea2. Latest consensus state: current_term: 1 leader_uuid: "cff23b1581dc4db5ad5b824e8969aea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff23b1581dc4db5ad5b824e8969aea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } } }
I20260620 23:05:51.040371 10567 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:51.040303 10568 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cff23b1581dc4db5ad5b824e8969aea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff23b1581dc4db5ad5b824e8969aea2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40459 } } }
I20260620 23:05:51.040560 10568 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:51.040599 10573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:51.041173 10573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 23:05:51 dist-test-slave-qgtk krb5kdc[10481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:05:51.042924 10573 catalog_manager.cc:1347] Generated new cluster ID: 4ac6762dd7bd4538aea88c4a2bb7b752
I20260620 23:05:51.042976 10573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:51.046496 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:51.034583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:45595 (local address 127.0.0.1:40459)
0620 23:05:51.034833 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:51.034842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:51.034871 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:05:51.035425 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:51.035433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:51.035514 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:51.035696 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:51.036452 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:51.037437 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:51.038368 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:51.039023 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:51.039148 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:51.042551 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:51.042555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:51.042556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:51.042569 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:51.044659 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:51.045179 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:51.045182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:51.045183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:51.045283 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:51.045551 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:51.045552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:51.045552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:51.045837 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:51.045989 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:51.046172 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 23:05:51.046273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260620 23:05:51.047499 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:51.061654 10573 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:51.062427 10573 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:51.062889 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:51.050519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:41281 (local address 127.0.0.1:40459)
0620 23:05:51.050567 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:51.050569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:51.054381 (+ 3812us) server_negotiation.cc:407] Connection header received
0620 23:05:51.054425 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:51.054426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:51.054440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:51.054518 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:51.055348 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:51.056189 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:51.057291 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:51.057629 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:51.057705 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:51.059354 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:51.059356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:51.059356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:51.059364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:51.061197 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:51.061873 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:51.061874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:51.061874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:51.061970 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:51.062227 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:51.062228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:51.062228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:51.062343 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:51.062457 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:51.062705 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:05:51.062774 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 23:05:51.065752 10573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cff23b1581dc4db5ad5b824e8969aea2: Generated new TSK 0
I20260620 23:05:51.065986 10573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:51.071309 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10494
W20260620 23:05:51.075558 10479 connection.cc:535] client connection to 127.0.0.1:40459 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40459): Connection reset by peer (error 104)
2026-06-20T23:05:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:51 dist-test-slave-qgtk krb5kdc[10593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:51 dist-test-slave-qgtk krb5kdc[10593](info): set up 2 sockets
Jun 20 23:05:51 dist-test-slave-qgtk krb5kdc[10593](info): commencing operation
krb5kdc: starting...
W20260620 23:05:52.122453 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:52 dist-test-slave-qgtk krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:52.249361 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:52.334704 10606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:52.334929 10606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:52.334954 10606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:52.336107 10606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:52.336148 10606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:52.336161 10606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:52.336174 10606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:52.336186 10606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:52.337647 10606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:52.337848 10606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:52.338028 10606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:52.343098 10614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:52.343118 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:52.343324 10612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:52.343269 10606 server_base.cc:1034] running on GCE node
I20260620 23:05:52.343586 10606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:52.343940 10606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:52.345206 10606 hybrid_clock.cc:648] HybridClock initialized: now 1781996752345176 us; error 39 us; skew 500 ppm
Jun 20 23:05:52 dist-test-slave-qgtk krb5kdc[10593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:52.347967 10606 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:52.348446 10606 webserver.cc:460] Webserver started at http://127.0.0.1:36777/ using document root <none> and password file <none>
I20260620 23:05:52.348685 10606 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:52.348748 10606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:52.348922 10606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:52.350010 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "0de15d2a4b3943ed8426152ec8edfbab"
format_stamp: "Formatted at 2026-06-20 23:05:52 on dist-test-slave-qgtk"
server_key: "6fedb9ab1fb569c9aa18207fe0c5e039"
server_key_iv: "c1ead86a412c1c42c955adedc588aeba"
server_key_version: "encryptionkey@0"
I20260620 23:05:52.350447 10606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "0de15d2a4b3943ed8426152ec8edfbab"
format_stamp: "Formatted at 2026-06-20 23:05:52 on dist-test-slave-qgtk"
server_key: "6fedb9ab1fb569c9aa18207fe0c5e039"
server_key_iv: "c1ead86a412c1c42c955adedc588aeba"
server_key_version: "encryptionkey@0"
I20260620 23:05:52.351910 10606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:52.352764 10621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:52.353050 10606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:52.353147 10606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "0de15d2a4b3943ed8426152ec8edfbab"
format_stamp: "Formatted at 2026-06-20 23:05:52 on dist-test-slave-qgtk"
server_key: "6fedb9ab1fb569c9aa18207fe0c5e039"
server_key_iv: "c1ead86a412c1c42c955adedc588aeba"
server_key_version: "encryptionkey@0"
I20260620 23:05:52.353227 10606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:52.379738 10606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:52.382238 10606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:52.382408 10606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:52.386824 10606 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39273
I20260620 23:05:52.386862 10673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39273 every 8 connection(s)
I20260620 23:05:52.387240 10606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:52.388229 10674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:52.390717 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab: Bootstrap starting.
I20260620 23:05:52.391301 10674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:52.391561 10674 log.cc:824] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:52.392320 10674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab: No bootstrap required, opened a new log
I20260620 23:05:52.394009 10674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de15d2a4b3943ed8426152ec8edfbab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39273 } }
I20260620 23:05:52.394232 10674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:52.394265 10674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0de15d2a4b3943ed8426152ec8edfbab, State: Initialized, Role: FOLLOWER
I20260620 23:05:52.394405 10674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de15d2a4b3943ed8426152ec8edfbab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39273 } }
I20260620 23:05:52.394495 10674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:52.394519 10674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:52.394555 10674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:52.394943 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10606
I20260620 23:05:52.395079 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:52.395315 21769 external_mini_cluster.cc:1428] Setting key 45c79381359f43e380320a55caefca13
I20260620 23:05:52.395325 10674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de15d2a4b3943ed8426152ec8edfbab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39273 } }
I20260620 23:05:52.395433 10674 leader_election.cc:302] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0de15d2a4b3943ed8426152ec8edfbab; no voters:
I20260620 23:05:52.395612 10674 leader_election.cc:288] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:52.395651 10677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:52.396044 10674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:52.396195 10677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [term 1 LEADER]: Becoming Leader. State: Replica: 0de15d2a4b3943ed8426152ec8edfbab, State: Running, Role: LEADER
I20260620 23:05:52.396323 10677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de15d2a4b3943ed8426152ec8edfbab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39273 } }
I20260620 23:05:52.396697 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0de15d2a4b3943ed8426152ec8edfbab. Latest consensus state: current_term: 1 leader_uuid: "0de15d2a4b3943ed8426152ec8edfbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de15d2a4b3943ed8426152ec8edfbab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39273 } } }
I20260620 23:05:52.397118 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:52.397034 10679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0de15d2a4b3943ed8426152ec8edfbab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de15d2a4b3943ed8426152ec8edfbab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39273 } } }
I20260620 23:05:52.397526 10679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:52.397557 10688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:52.398300 10688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:52.400019 10688 catalog_manager.cc:1347] Generated new cluster ID: d89df165b95249578f87d95abb197fc5
I20260620 23:05:52.400079 10688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:52.401927 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:52.395899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:40851 (local address 127.0.0.1:39273)
0620 23:05:52.396073 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:52.396084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:52.396122 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:05:52.396247 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:52.396254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:52.396331 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:52.396477 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:52.396820 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:52.398353 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:52.399445 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:52.400051 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:52.400159 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:52.401196 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:52.401200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:52.401200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:52.401417 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:52.401457 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:52.401473 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:52.401564 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:52.401671 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 23:05:52.401701 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260620 23:05:52.402985 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:52.410346 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:52.406412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:50257 (local address 127.0.0.1:39273)
0620 23:05:52.406458 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:52.406460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:52.406616 (+ 156us) server_negotiation.cc:407] Connection header received
0620 23:05:52.406669 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:52.406670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:52.406682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:52.406739 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:52.406882 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:52.407720 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:52.408787 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:52.409165 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:52.409231 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:52.410025 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:52.410027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:52.410027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:52.410032 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:52.410050 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:52.410055 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:52.410156 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:52.410237 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 23:05:52.410251 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:05:52.420706 10688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:52.421314 10688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:52.427029 10688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0de15d2a4b3943ed8426152ec8edfbab: Generated new TSK 0
I20260620 23:05:52.427248 10688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:52.429984 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10606
W20260620 23:05:52.434054 10591 connection.cc:535] client connection to 127.0.0.1:39273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39273): Connection reset by peer (error 104)
2026-06-20T23:05:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:52 dist-test-slave-qgtk krb5kdc[10705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:52 dist-test-slave-qgtk krb5kdc[10705](info): set up 2 sockets
Jun 20 23:05:52 dist-test-slave-qgtk krb5kdc[10705](info): commencing operation
krb5kdc: starting...
W20260620 23:05:53.494450 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:53 dist-test-slave-qgtk krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:05:53.613204 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:53.696940 10718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:53.697273 10718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:53.697300 10718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:53.698967 10718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:53.699033 10718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:53.699062 10718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:53.699102 10718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:53.699120 10718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:53.700881 10718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:53.701179 10718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:53.701424 10718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:53.706470 10724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:53.706449 10726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:53.706668 10723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:53.706697 10718 server_base.cc:1034] running on GCE node
I20260620 23:05:53.706943 10718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:53.707232 10718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:53.708400 10718 hybrid_clock.cc:648] HybridClock initialized: now 1781996753708389 us; error 43 us; skew 500 ppm
Jun 20 23:05:53 dist-test-slave-qgtk krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:53.710980 10718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:05:53.711470 10718 webserver.cc:460] Webserver started at http://127.0.0.1:36979/ using document root <none> and password file <none>
I20260620 23:05:53.711696 10718 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:53.711761 10718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:53.711902 10718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:53.712989 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "0bec1ba45cc5432fae84b40dc87913ba"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-qgtk"
server_key: "a47d7434162fdac9889d0faac48279ec"
server_key_iv: "70de066235358414e3f89af308824bd3"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.713349 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "0bec1ba45cc5432fae84b40dc87913ba"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-qgtk"
server_key: "a47d7434162fdac9889d0faac48279ec"
server_key_iv: "70de066235358414e3f89af308824bd3"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.714752 10718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:53.715487 10733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:53.715806 10718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:53.715925 10718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "0bec1ba45cc5432fae84b40dc87913ba"
format_stamp: "Formatted at 2026-06-20 23:05:53 on dist-test-slave-qgtk"
server_key: "a47d7434162fdac9889d0faac48279ec"
server_key_iv: "70de066235358414e3f89af308824bd3"
server_key_version: "encryptionkey@0"
I20260620 23:05:53.716002 10718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:53.719367 10718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:53.721509 10718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:53.721704 10718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:53.725912 10718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37323
I20260620 23:05:53.725958 10785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37323 every 8 connection(s)
I20260620 23:05:53.726367 10718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:53.727353 10786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:53.729527 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10718
I20260620 23:05:53.729661 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:53.729856 21769 external_mini_cluster.cc:1428] Setting key 8e575e1e3c05f0e3a2b72580eea853c6
I20260620 23:05:53.729866 10786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba: Bootstrap starting.
I20260620 23:05:53.730607 10786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:53.730938 10786 log.cc:824] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:53.731781 10786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba: No bootstrap required, opened a new log
I20260620 23:05:53.732658 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.730443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:56825 (local address 127.0.0.1:37323)
0620 23:05:53.730635 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:53.730645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:53.730684 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:05:53.731276 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:53.731284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:53.731329 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:53.731476 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:53.732232 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:53.732236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:53.732236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:53.732240 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:53.732288 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:53.732303 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:53.732395 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:53.732425 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 23:05:53.732470 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260620 23:05:53.733700 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:53.734321 10786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec1ba45cc5432fae84b40dc87913ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37323 } }
I20260620 23:05:53.734505 10786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:53.734560 10786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bec1ba45cc5432fae84b40dc87913ba, State: Initialized, Role: FOLLOWER
I20260620 23:05:53.734691 10786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec1ba45cc5432fae84b40dc87913ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37323 } }
I20260620 23:05:53.734781 10786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:53.734827 10786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:53.734889 10786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:53.735698 10786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec1ba45cc5432fae84b40dc87913ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37323 } }
I20260620 23:05:53.735816 10786 leader_election.cc:302] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bec1ba45cc5432fae84b40dc87913ba; no voters:
I20260620 23:05:53.735966 10786 leader_election.cc:288] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:53.736012 10796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:53.736193 10796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [term 1 LEADER]: Becoming Leader. State: Replica: 0bec1ba45cc5432fae84b40dc87913ba, State: Running, Role: LEADER
I20260620 23:05:53.736210 10786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:53.736306 10796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec1ba45cc5432fae84b40dc87913ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37323 } }
I20260620 23:05:53.736593 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:53.735335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:56305 (local address 127.0.0.1:37323)
0620 23:05:53.735383 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:53.735385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:53.735455 (+ 70us) server_negotiation.cc:407] Connection header received
0620 23:05:53.735530 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:53.735530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:53.735534 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:53.735578 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:53.736232 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:53.736233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:53.736233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:53.736238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:53.736252 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:53.736256 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:53.736286 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:53.736482 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:05:53.736511 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:05:53.736742 10796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bec1ba45cc5432fae84b40dc87913ba. Latest consensus state: current_term: 1 leader_uuid: "0bec1ba45cc5432fae84b40dc87913ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec1ba45cc5432fae84b40dc87913ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37323 } } }
I20260620 23:05:53.736824 10796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:53.736980 10797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bec1ba45cc5432fae84b40dc87913ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bec1ba45cc5432fae84b40dc87913ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37323 } } }
I20260620 23:05:53.737066 10797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:53.737517 10804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:53.738008 10804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:53.739472 10804 catalog_manager.cc:1347] Generated new cluster ID: 07ceffacc5494dfba92f855a2fc83d54
I20260620 23:05:53.739526 10804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:53.759292 10804 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:53.759500 10804 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:53.763438 10804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bec1ba45cc5432fae84b40dc87913ba: Generated new TSK 0
I20260620 23:05:53.763693 10804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:53.769656 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10718
2026-06-20T23:05:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:53 dist-test-slave-qgtk krb5kdc[10817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:53 dist-test-slave-qgtk krb5kdc[10817](info): set up 2 sockets
Jun 20 23:05:53 dist-test-slave-qgtk krb5kdc[10817](info): commencing operation
krb5kdc: starting...
W20260620 23:05:54.822369 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:54 dist-test-slave-qgtk krb5kdc[10817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:05:54.949913 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:37629
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:37629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:55.040197 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:55.040488 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:55.040558 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:55.042315 10830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:55.042407 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:55.042438 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:55.042480 10830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:55.042506 10830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:55.044263 10830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:37629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:37629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:55.044586 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:55.044839 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:55.050284 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:55.050325 10830 server_base.cc:1034] running on GCE node
W20260620 23:05:55.050295 10836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:55.050499 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:55.050987 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:55.051267 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:55.052454 10830 hybrid_clock.cc:648] HybridClock initialized: now 1781996755052397 us; error 77 us; skew 500 ppm
Jun 20 23:05:55 dist-test-slave-qgtk krb5kdc[10817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:55.055224 10830 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:55.055785 10830 webserver.cc:460] Webserver started at http://127.21.66.126:36915/ using document root <none> and password file <none>
I20260620 23:05:55.056053 10830 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:55.056123 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:55.056288 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:55.057469 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "0b5f34b73b314c0a9098cc2484853ff9"
format_stamp: "Formatted at 2026-06-20 23:05:55 on dist-test-slave-qgtk"
server_key: "24354e056a3bae534ad75f434d1b9dc7"
server_key_iv: "233ce78f91da72a94958ec434145f76c"
server_key_version: "encryptionkey@0"
I20260620 23:05:55.057866 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "0b5f34b73b314c0a9098cc2484853ff9"
format_stamp: "Formatted at 2026-06-20 23:05:55 on dist-test-slave-qgtk"
server_key: "24354e056a3bae534ad75f434d1b9dc7"
server_key_iv: "233ce78f91da72a94958ec434145f76c"
server_key_version: "encryptionkey@0"
I20260620 23:05:55.059562 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260620 23:05:55.060428 10845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:55.060643 10830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:55.060756 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "0b5f34b73b314c0a9098cc2484853ff9"
format_stamp: "Formatted at 2026-06-20 23:05:55 on dist-test-slave-qgtk"
server_key: "24354e056a3bae534ad75f434d1b9dc7"
server_key_iv: "233ce78f91da72a94958ec434145f76c"
server_key_version: "encryptionkey@0"
I20260620 23:05:55.060837 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:55.073225 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:55.075886 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:55.076074 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:55.080580 10830 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:37629
I20260620 23:05:55.080616 10897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:37629 every 8 connection(s)
I20260620 23:05:55.081003 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:55.081880 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:55.084254 10898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9: Bootstrap starting.
I20260620 23:05:55.084852 10898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:55.085134 10898 log.cc:824] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:55.085597 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10830
I20260620 23:05:55.085740 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:55.085948 21769 external_mini_cluster.cc:1428] Setting key 0e1f642f4011847960fd75696731b7ed
I20260620 23:05:55.086158 10898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9: No bootstrap required, opened a new log
I20260620 23:05:55.088225 10898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5f34b73b314c0a9098cc2484853ff9" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 37629 } }
I20260620 23:05:55.088383 10898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:55.088445 10898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5f34b73b314c0a9098cc2484853ff9, State: Initialized, Role: FOLLOWER
I20260620 23:05:55.088591 10898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5f34b73b314c0a9098cc2484853ff9" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 37629 } }
I20260620 23:05:55.088685 10898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:55.088730 10898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:55.088781 10898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:55.089327 10898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5f34b73b314c0a9098cc2484853ff9" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 37629 } }
I20260620 23:05:55.089442 10898 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5f34b73b314c0a9098cc2484853ff9; no voters:
I20260620 23:05:55.089625 10898 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:55.089664 10903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:55.089825 10903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 0b5f34b73b314c0a9098cc2484853ff9, State: Running, Role: LEADER
I20260620 23:05:55.089897 10898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:55.089915 10903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5f34b73b314c0a9098cc2484853ff9" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 37629 } }
I20260620 23:05:55.090325 10905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5f34b73b314c0a9098cc2484853ff9. Latest consensus state: current_term: 1 leader_uuid: "0b5f34b73b314c0a9098cc2484853ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5f34b73b314c0a9098cc2484853ff9" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 37629 } } }
I20260620 23:05:55.090410 10905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [sys.catalog]: This master's current role is: LEADER
W20260620 23:05:55.091691 10919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:55.091750 10919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:55.091769 10904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5f34b73b314c0a9098cc2484853ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5f34b73b314c0a9098cc2484853ff9" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 37629 } } }
I20260620 23:05:55.091807 10910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:55.091835 10904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:55.091971 10910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:55.094172 10910 catalog_manager.cc:1347] Generated new cluster ID: 1dcc4f39b45b4e038351e80f665cdd3c
I20260620 23:05:55.094244 10910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 23:05:55 dist-test-slave-qgtk krb5kdc[10817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:55.098672 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.086577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47503 (local address 127.21.66.126:37629)
0620 23:05:55.086772 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.086782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.086816 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:05:55.087488 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.087496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.087556 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.087743 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:55.088464 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.090101 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.091399 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.092016 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.092144 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.094971 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:55.094976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:55.094976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:55.094989 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:55.096702 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.097114 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.097117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.097117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.097223 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.097452 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.097453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.097453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.097710 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:55.098343 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.098422 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.098516 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260620 23:05:55.099668 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:55.107620 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:55.101124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57231 (local address 127.21.66.126:37629)
0620 23:05:55.101159 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:55.101161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:55.101169 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:05:55.101211 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:55.101212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:55.101222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:55.101276 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:55.101793 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.102619 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.103378 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:55.103620 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:55.103705 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:55.104961 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:55.104962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:55.104962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:55.104970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:55.106410 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.106770 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.106771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.106771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.106835 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:55.107050 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:55.107051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:55.107051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:55.107163 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:55.107282 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:55.107420 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 23:05:55.107496 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:05:55.110047 10910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:55.110597 10910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:55.119263 10910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5f34b73b314c0a9098cc2484853ff9: Generated new TSK 0
I20260620 23:05:55.119562 10910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:55.124013 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10830
W20260620 23:05:55.128873 10815 connection.cc:535] client connection to 127.21.66.126:37629 recv error: Network error: failed to read from TLS socket (remote: 127.21.66.126:37629): Connection reset by peer (error 104)
2026-06-20T23:05:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:55 dist-test-slave-qgtk krb5kdc[10929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:55 dist-test-slave-qgtk krb5kdc[10929](info): set up 2 sockets
Jun 20 23:05:55 dist-test-slave-qgtk krb5kdc[10929](info): commencing operation
krb5kdc: starting...
W20260620 23:05:56.254478 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:56 dist-test-slave-qgtk krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:05:56.385469 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39973
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:56.467166 10942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:56.467425 10942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:56.467466 10942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:56.468631 10942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:56.468703 10942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:56.468727 10942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:56.468756 10942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:56.468787 10942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:56.470326 10942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:56.470624 10942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:56.470868 10942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:56.475914 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:56.476183 10942 server_base.cc:1034] running on GCE node
W20260620 23:05:56.476003 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:56.475996 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:56.476570 10942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:56.476894 10942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:56.478082 10942 hybrid_clock.cc:648] HybridClock initialized: now 1781996756478019 us; error 77 us; skew 500 ppm
Jun 20 23:05:56 dist-test-slave-qgtk krb5kdc[10929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:56.485792 10942 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:56.486361 10942 webserver.cc:460] Webserver started at http://127.21.66.126:44035/ using document root <none> and password file <none>
I20260620 23:05:56.486620 10942 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:56.486697 10942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:56.486860 10942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:56.487986 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "25c4aba058834d8a829e274f8a4722e6"
format_stamp: "Formatted at 2026-06-20 23:05:56 on dist-test-slave-qgtk"
server_key: "9a57a388e00048d25cbd2039e75ec118"
server_key_iv: "3c8ea91ce847cb2ab52412b5ea14582d"
server_key_version: "encryptionkey@0"
I20260620 23:05:56.488353 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "25c4aba058834d8a829e274f8a4722e6"
format_stamp: "Formatted at 2026-06-20 23:05:56 on dist-test-slave-qgtk"
server_key: "9a57a388e00048d25cbd2039e75ec118"
server_key_iv: "3c8ea91ce847cb2ab52412b5ea14582d"
server_key_version: "encryptionkey@0"
I20260620 23:05:56.489881 10942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:56.490801 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:56.491003 10942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 23:05:56.491109 10942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "25c4aba058834d8a829e274f8a4722e6"
format_stamp: "Formatted at 2026-06-20 23:05:56 on dist-test-slave-qgtk"
server_key: "9a57a388e00048d25cbd2039e75ec118"
server_key_iv: "3c8ea91ce847cb2ab52412b5ea14582d"
server_key_version: "encryptionkey@0"
I20260620 23:05:56.491195 10942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:56.510064 10942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:56.512447 10942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:56.512629 10942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:56.516870 10942 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:39973
I20260620 23:05:56.516919 11009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:39973 every 8 connection(s)
I20260620 23:05:56.517314 10942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:56.518325 11010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:56.520606 11010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6: Bootstrap starting.
I20260620 23:05:56.520979 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 10942
I20260620 23:05:56.521086 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:56.521247 11010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:56.521312 21769 external_mini_cluster.cc:1428] Setting key b07d89a2ca2a62f876970a13cd74eb32
I20260620 23:05:56.521554 11010 log.cc:824] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:56.522410 11010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6: No bootstrap required, opened a new log
I20260620 23:05:56.524556 11010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4aba058834d8a829e274f8a4722e6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39973 } }
I20260620 23:05:56.524731 11010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:56.524761 11010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c4aba058834d8a829e274f8a4722e6, State: Initialized, Role: FOLLOWER
I20260620 23:05:56.524879 11010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4aba058834d8a829e274f8a4722e6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39973 } }
I20260620 23:05:56.524951 11010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:56.524981 11010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:56.525024 11010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:56.525578 11010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4aba058834d8a829e274f8a4722e6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39973 } }
I20260620 23:05:56.525705 11010 leader_election.cc:302] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c4aba058834d8a829e274f8a4722e6; no voters:
I20260620 23:05:56.525878 11010 leader_election.cc:288] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:56.525905 11015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:56.526113 11015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [term 1 LEADER]: Becoming Leader. State: Replica: 25c4aba058834d8a829e274f8a4722e6, State: Running, Role: LEADER
I20260620 23:05:56.526204 11010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:56.526293 11015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4aba058834d8a829e274f8a4722e6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39973 } }
I20260620 23:05:56.527294 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:56.521878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48715 (local address 127.21.66.126:39973)
0620 23:05:56.522052 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:56.522062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:56.522101 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:05:56.522627 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:56.522635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:56.522685 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:56.522813 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:56.523094 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.524008 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.525135 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.525573 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.525682 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:56.526554 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:56.526558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:56.526559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:56.526811 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:56.526851 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:56.526868 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:56.526955 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:56.527067 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:05:56.527098 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260620 23:05:56.527314 11016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c4aba058834d8a829e274f8a4722e6. Latest consensus state: current_term: 1 leader_uuid: "25c4aba058834d8a829e274f8a4722e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4aba058834d8a829e274f8a4722e6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39973 } } }
I20260620 23:05:56.527426 11016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:56.527797 11017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c4aba058834d8a829e274f8a4722e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c4aba058834d8a829e274f8a4722e6" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39973 } } }
I20260620 23:05:56.528005 11017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6 [sys.catalog]: This master's current role is: LEADER
W20260620 23:05:56.528064 11031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:05:56.528122 11031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:05:56.528280 11030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:56.528440 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:56.528532 11030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:56.530408 11030 catalog_manager.cc:1347] Generated new cluster ID: bf748465c07f454e9468a287fbaf90ba
I20260620 23:05:56.530467 11030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:56.533553 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:56.529945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33213 (local address 127.21.66.126:39973)
0620 23:05:56.529985 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:56.529987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:56.530053 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:05:56.530149 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:56.530150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:56.530160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:56.530235 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:56.530485 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.531218 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.532220 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:56.532532 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:56.532606 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:56.533298 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:56.533300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:56.533300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:56.533305 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:56.533321 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:56.533326 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:56.533390 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:56.533467 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:05:56.533477 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:05:56.542934 11030 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:56.543458 11030 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:56.545884 11030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25c4aba058834d8a829e274f8a4722e6: Generated new TSK 0
I20260620 23:05:56.546149 11030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:56.549790 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 10942
W20260620 23:05:56.554482 10927 connection.cc:535] client connection to 127.21.66.126:39973 recv error: Network error: failed to read from TLS socket (remote: 127.21.66.126:39973): Connection reset by peer (error 104)
2026-06-20T23:05:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:56 dist-test-slave-qgtk krb5kdc[11041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:56 dist-test-slave-qgtk krb5kdc[11041](info): set up 2 sockets
Jun 20 23:05:56 dist-test-slave-qgtk krb5kdc[11041](info): commencing operation
krb5kdc: starting...
W20260620 23:05:57.634877 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:57 dist-test-slave-qgtk krb5kdc[11041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:05:57.763249 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40459
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:40459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:57.844157 11054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:57.844427 11054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:57.844466 11054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:57.845836 11054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:57.845914 11054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:57.845939 11054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:57.845960 11054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:57.845989 11054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:57.847549 11054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:40459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:57.847844 11054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:57.848084 11054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:57.853192 11060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:57.853165 11062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:57.853144 11059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:57.853200 11054 server_base.cc:1034] running on GCE node
I20260620 23:05:57.853806 11054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:57.854067 11054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:57.855252 11054 hybrid_clock.cc:648] HybridClock initialized: now 1781996757855240 us; error 32 us; skew 500 ppm
Jun 20 23:05:57 dist-test-slave-qgtk krb5kdc[11041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996757, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:57.858074 11054 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:57.858783 11054 webserver.cc:460] Webserver started at http://127.21.66.126:43647/ using document root <none> and password file <none>
I20260620 23:05:57.859037 11054 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:57.859090 11054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:57.859232 11054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:57.860430 11054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "d0e0bca4f86f493dbfce121bf9f27f51"
format_stamp: "Formatted at 2026-06-20 23:05:57 on dist-test-slave-qgtk"
server_key: "2d4b4a76f20c8cc55e43a82056d3df94"
server_key_iv: "57a2f6465c475d63948603f21e2df5cc"
server_key_version: "encryptionkey@0"
I20260620 23:05:57.860823 11054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "d0e0bca4f86f493dbfce121bf9f27f51"
format_stamp: "Formatted at 2026-06-20 23:05:57 on dist-test-slave-qgtk"
server_key: "2d4b4a76f20c8cc55e43a82056d3df94"
server_key_iv: "57a2f6465c475d63948603f21e2df5cc"
server_key_version: "encryptionkey@0"
I20260620 23:05:57.862438 11054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:57.863360 11069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:57.863538 11054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:57.863654 11054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "d0e0bca4f86f493dbfce121bf9f27f51"
format_stamp: "Formatted at 2026-06-20 23:05:57 on dist-test-slave-qgtk"
server_key: "2d4b4a76f20c8cc55e43a82056d3df94"
server_key_iv: "57a2f6465c475d63948603f21e2df5cc"
server_key_version: "encryptionkey@0"
I20260620 23:05:57.863732 11054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:57.867166 11054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:57.869261 11054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:57.869413 11054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:57.873667 11054 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:40459
I20260620 23:05:57.873709 11121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:40459 every 8 connection(s)
I20260620 23:05:57.874058 11054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:57.875130 11122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:57.877310 11122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51: Bootstrap starting.
I20260620 23:05:57.877846 11122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:57.878201 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11054
I20260620 23:05:57.878209 11122 log.cc:824] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:57.878322 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:57.878511 21769 external_mini_cluster.cc:1428] Setting key 0761605cd826a6ef7469820a7cf9f5be
I20260620 23:05:57.879083 11122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51: No bootstrap required, opened a new log
I20260620 23:05:57.881045 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:57.879015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55001 (local address 127.21.66.126:40459)
0620 23:05:57.879204 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:57.879213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:57.879247 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:05:57.879665 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:57.879672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:57.879698 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:57.879851 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:57.880582 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:57.880586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:57.880586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:57.880590 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:57.880696 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:57.880708 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:57.880806 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:57.880844 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 23:05:57.880885 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260620 23:05:57.882212 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:57.882227 11122 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40459 } }
I20260620 23:05:57.882459 11122 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:57.882519 11122 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e0bca4f86f493dbfce121bf9f27f51, State: Initialized, Role: FOLLOWER
I20260620 23:05:57.882647 11122 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40459 } }
I20260620 23:05:57.882766 11122 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:57.882813 11122 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:57.882866 11122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:05:57.883597 11122 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40459 } }
I20260620 23:05:57.883735 11122 leader_election.cc:302] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0e0bca4f86f493dbfce121bf9f27f51; no voters:
I20260620 23:05:57.883983 11122 leader_election.cc:288] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:57.884020 11132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:57.884258 11132 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [term 1 LEADER]: Becoming Leader. State: Replica: d0e0bca4f86f493dbfce121bf9f27f51, State: Running, Role: LEADER
I20260620 23:05:57.884357 11122 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:57.884383 11132 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40459 } }
I20260620 23:05:57.884780 11132 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e0bca4f86f493dbfce121bf9f27f51. Latest consensus state: current_term: 1 leader_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40459 } } }
I20260620 23:05:57.885006 11132 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:57.885217 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:57.883781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60053 (local address 127.21.66.126:40459)
0620 23:05:57.883814 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:57.883815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:57.883947 (+ 132us) server_negotiation.cc:407] Connection header received
0620 23:05:57.884039 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:57.884040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:57.884046 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:57.884101 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:57.884890 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:57.884891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:57.884891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:05:57.884908 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:57.884925 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:05:57.884930 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:57.884968 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:57.885127 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:05:57.885142 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:05:57.884932 11133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e0bca4f86f493dbfce121bf9f27f51" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 40459 } } }
I20260620 23:05:57.885367 11133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:57.885537 11141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:57.886089 11141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:57.887687 11141 catalog_manager.cc:1347] Generated new cluster ID: 3955d5a5842d4701be13f3143c8ca6e4
I20260620 23:05:57.887744 11141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:57.896885 11141 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:57.897075 11141 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:57.901660 11141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0e0bca4f86f493dbfce121bf9f27f51: Generated new TSK 0
I20260620 23:05:57.901904 11141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:57.903923 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11054
2026-06-20T23:05:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:57 dist-test-slave-qgtk krb5kdc[11153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:57 dist-test-slave-qgtk krb5kdc[11153](info): set up 2 sockets
Jun 20 23:05:57 dist-test-slave-qgtk krb5kdc[11153](info): commencing operation
krb5kdc: starting...
W20260620 23:05:58.950368 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:05:59 dist-test-slave-qgtk krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:05:59.076536 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:43645
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:05:59.157353 11166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:05:59.157612 11166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:05:59.157651 11166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:05:59.158823 11166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:05:59.158895 11166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:05:59.158921 11166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:05:59.158949 11166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:05:59.158980 11166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:05:59.160523 11166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:05:59.160813 11166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:05:59.161056 11166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:05:59.165999 11172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:59.166050 11171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:05:59.166050 11174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:05:59.166314 11166 server_base.cc:1034] running on GCE node
I20260620 23:05:59.166606 11166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:05:59.166900 11166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:05:59.168077 11166 hybrid_clock.cc:648] HybridClock initialized: now 1781996759168065 us; error 32 us; skew 500 ppm
Jun 20 23:05:59 dist-test-slave-qgtk krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:05:59.170980 11166 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:05:59.171460 11166 webserver.cc:460] Webserver started at http://127.21.66.126:34525/ using document root <none> and password file <none>
I20260620 23:05:59.171646 11166 fs_manager.cc:359] Metadata directory not provided
I20260620 23:05:59.171686 11166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:05:59.171774 11166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:05:59.172786 11166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "564df92f1b7145c68f3e8c49d2bda8e1"
format_stamp: "Formatted at 2026-06-20 23:05:59 on dist-test-slave-qgtk"
server_key: "7f61537d2d75173e04d523c2ed5fe4ee"
server_key_iv: "d6c28fa86fe0d4e2f7ac1d9073bb0068"
server_key_version: "encryptionkey@0"
I20260620 23:05:59.173553 11166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "564df92f1b7145c68f3e8c49d2bda8e1"
format_stamp: "Formatted at 2026-06-20 23:05:59 on dist-test-slave-qgtk"
server_key: "7f61537d2d75173e04d523c2ed5fe4ee"
server_key_iv: "d6c28fa86fe0d4e2f7ac1d9073bb0068"
server_key_version: "encryptionkey@0"
I20260620 23:05:59.175127 11166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:05:59.175897 11181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:05:59.176172 11166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:05:59.176246 11166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "564df92f1b7145c68f3e8c49d2bda8e1"
format_stamp: "Formatted at 2026-06-20 23:05:59 on dist-test-slave-qgtk"
server_key: "7f61537d2d75173e04d523c2ed5fe4ee"
server_key_iv: "d6c28fa86fe0d4e2f7ac1d9073bb0068"
server_key_version: "encryptionkey@0"
I20260620 23:05:59.176302 11166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:05:59.192056 11166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:05:59.194706 11166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:05:59.194844 11166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:05:59.199517 11166 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:43645
I20260620 23:05:59.199548 11233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:43645 every 8 connection(s)
I20260620 23:05:59.199932 11166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:05:59.200802 11234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:05:59.201496 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11166
I20260620 23:05:59.201620 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:05:59.201817 21769 external_mini_cluster.cc:1428] Setting key 554b7957075f3d142eff09e8c775cec4
I20260620 23:05:59.204062 11234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1: Bootstrap starting.
I20260620 23:05:59.204797 11234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1: Neither blocks nor log segments found. Creating new log.
I20260620 23:05:59.205112 11234 log.cc:824] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:05:59.205873 11234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1: No bootstrap required, opened a new log
I20260620 23:05:59.208222 11234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43645 } }
I20260620 23:05:59.208360 11234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:05:59.208382 11234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564df92f1b7145c68f3e8c49d2bda8e1, State: Initialized, Role: FOLLOWER
I20260620 23:05:59.208526 11234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43645 } }
I20260620 23:05:59.208602 11234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:05:59.208653 11234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:05:59.208712 11234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:05:59 dist-test-slave-qgtk krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:05:59.209424 11234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43645 } }
I20260620 23:05:59.209561 11234 leader_election.cc:302] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 564df92f1b7145c68f3e8c49d2bda8e1; no voters:
I20260620 23:05:59.209755 11234 leader_election.cc:288] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:05:59.209789 11239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:05:59.209905 11239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [term 1 LEADER]: Becoming Leader. State: Replica: 564df92f1b7145c68f3e8c49d2bda8e1, State: Running, Role: LEADER
I20260620 23:05:59.209990 11239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43645 } }
I20260620 23:05:59.210023 11234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:05:59.210355 11239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43645 } } }
I20260620 23:05:59.210427 11239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:59.210546 11239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 564df92f1b7145c68f3e8c49d2bda8e1. Latest consensus state: current_term: 1 leader_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "564df92f1b7145c68f3e8c49d2bda8e1" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 43645 } } }
I20260620 23:05:59.210600 11239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:05:59.210738 11244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:05:59.211280 11244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:05:59.213431 11244 catalog_manager.cc:1347] Generated new cluster ID: f93e64d54d674f14bdc3898737e71f3b
I20260620 23:05:59.213479 11244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:05:59.213716 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:59.202471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52103 (local address 127.21.66.126:43645)
0620 23:05:59.202682 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:59.202691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:59.202727 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:05:59.203314 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:59.203324 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:59.203390 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:59.203550 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:59.204203 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:59.205531 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:59.206416 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:59.207034 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:59.207158 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:59.209939 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:59.209943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:59.209944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:59.209957 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:59.212104 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:59.212454 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:59.212456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:59.212457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:59.212562 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:59.212777 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:59.212778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:59.212778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:59.213088 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:59.213240 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:59.213370 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 23:05:59.213518 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260620 23:05:59.214670 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:05:59.219856 11244 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:05:59.220609 11244 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:05:59.223673 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:05:59.216222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55285 (local address 127.21.66.126:43645)
0620 23:05:59.216255 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:05:59.216256 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:05:59.216366 (+ 110us) server_negotiation.cc:407] Connection header received
0620 23:05:59.216422 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:05:59.216423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:05:59.216432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:05:59.216480 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:05:59.217116 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:59.217837 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:59.218901 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:05:59.219208 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:05:59.219277 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:05:59.220768 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:05:59.220769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:05:59.220770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:05:59.220777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:05:59.222407 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:59.222780 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:59.222781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:59.222781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:59.222840 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:05:59.223071 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:05:59.223071 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:05:59.223072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:05:59.223193 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:05:59.223263 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:05:59.223485 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 23:05:59.223578 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:05:59.232751 11244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 564df92f1b7145c68f3e8c49d2bda8e1: Generated new TSK 0
I20260620 23:05:59.233031 11244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:05:59.236740 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11166
2026-06-20T23:05:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:05:59 dist-test-slave-qgtk krb5kdc[11265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:05:59 dist-test-slave-qgtk krb5kdc[11265](info): set up 2 sockets
Jun 20 23:05:59 dist-test-slave-qgtk krb5kdc[11265](info): commencing operation
krb5kdc: starting...
W20260620 23:06:00.278265 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:06:00 dist-test-slave-qgtk krb5kdc[11265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:06:00.405725 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:35319
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:00.489599 11278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:00.489856 11278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:00.489893 11278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:00.491096 11278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:00.491180 11278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:00.491204 11278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:00.491223 11278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:00.491254 11278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:00.492755 11278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:35319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:00.493037 11278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:00.493275 11278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:00.498505 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:00.498750 11278 server_base.cc:1034] running on GCE node
W20260620 23:06:00.498554 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:00.498517 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:00.499076 11278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:00.499400 11278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:00.500563 11278 hybrid_clock.cc:648] HybridClock initialized: now 1781996760500543 us; error 44 us; skew 500 ppm
Jun 20 23:06:00 dist-test-slave-qgtk krb5kdc[11265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996760, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:00.503393 11278 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:06:00.503822 11278 webserver.cc:460] Webserver started at http://127.21.66.126:42529/ using document root <none> and password file <none>
I20260620 23:06:00.504058 11278 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:00.504104 11278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:00.504201 11278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:00.505306 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "55558ded87bc4debaa1d2dba3bcae4a7"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-qgtk"
server_key: "28f3ec610e244caae22797fdff920420"
server_key_iv: "9c1529569260eec2e23ee6ef118a112a"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.505622 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "55558ded87bc4debaa1d2dba3bcae4a7"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-qgtk"
server_key: "28f3ec610e244caae22797fdff920420"
server_key_iv: "9c1529569260eec2e23ee6ef118a112a"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.507186 11278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 23:06:00.508016 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:00.508231 11278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:06:00.508359 11278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "55558ded87bc4debaa1d2dba3bcae4a7"
format_stamp: "Formatted at 2026-06-20 23:06:00 on dist-test-slave-qgtk"
server_key: "28f3ec610e244caae22797fdff920420"
server_key_iv: "9c1529569260eec2e23ee6ef118a112a"
server_key_version: "encryptionkey@0"
I20260620 23:06:00.508471 11278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:00.518478 11278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:00.521353 11278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:00.521522 11278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:00.525427 11278 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:35319
I20260620 23:06:00.525497 11345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:35319 every 8 connection(s)
I20260620 23:06:00.525875 11278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:00.526849 11346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:00.529270 11346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7: Bootstrap starting.
I20260620 23:06:00.529901 11346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:00.530233 11346 log.cc:824] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:00.531013 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11278
I20260620 23:06:00.531136 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:06:00.531309 21769 external_mini_cluster.cc:1428] Setting key 02d9c64b240e6680c80dbdd7d5b82e0a
I20260620 23:06:00.531320 11346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7: No bootstrap required, opened a new log
I20260620 23:06:00.534651 11346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35319 } }
I20260620 23:06:00.534840 11346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:00.534873 11346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55558ded87bc4debaa1d2dba3bcae4a7, State: Initialized, Role: FOLLOWER
I20260620 23:06:00.535008 11346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35319 } }
I20260620 23:06:00.535092 11346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:06:00.535135 11346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:06:00.535233 11346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:00.535988 11346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35319 } }
I20260620 23:06:00.536103 11346 leader_election.cc:302] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55558ded87bc4debaa1d2dba3bcae4a7; no voters:
I20260620 23:06:00.536276 11346 leader_election.cc:288] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:06:00.536301 11351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:00.536550 11351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [term 1 LEADER]: Becoming Leader. State: Replica: 55558ded87bc4debaa1d2dba3bcae4a7, State: Running, Role: LEADER
I20260620 23:06:00.536562 11346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:00.536720 11351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35319 } }
I20260620 23:06:00.537914 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55558ded87bc4debaa1d2dba3bcae4a7. Latest consensus state: current_term: 1 leader_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35319 } } }
I20260620 23:06:00.538035 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:00.538344 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55558ded87bc4debaa1d2dba3bcae4a7" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 35319 } } }
W20260620 23:06:00.538674 11367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:06:00.538785 11367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:06:00.538709 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:00.538892 11366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:00.539165 11366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:00.539384 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:00.531861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35131 (local address 127.21.66.126:35319)
0620 23:06:00.532080 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:00.532093 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:00.532134 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:06:00.532949 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:00.532960 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:00.533033 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:00.533208 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:00.533432 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.535055 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.536217 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.536966 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.537103 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:00.538475 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:00.538479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:00.538480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:00.538683 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:00.538724 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:00.538740 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:00.538805 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:00.539021 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:06:00.539096 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260620 23:06:00.540453 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:06:00.540869 11366 catalog_manager.cc:1347] Generated new cluster ID: a63fd678f8294079afeccca914d9764a
I20260620 23:06:00.540951 11366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:00.545872 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:00.542017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39359 (local address 127.21.66.126:35319)
0620 23:06:00.542051 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:00.542053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:00.542325 (+ 272us) server_negotiation.cc:407] Connection header received
0620 23:06:00.542379 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:00.542380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:00.542393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:00.542468 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:00.542593 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.543427 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.544438 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:00.544788 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:00.544871 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:00.545583 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:00.545585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:00.545585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:00.545591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:00.545609 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:00.545614 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:00.545683 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:00.545729 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 23:06:00.545757 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:06:00.549511 11366 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:00.550001 11366 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:00.554347 11366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55558ded87bc4debaa1d2dba3bcae4a7: Generated new TSK 0
I20260620 23:06:00.554522 11366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:06:00.561687 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11278
2026-06-20T23:06:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:06:00 dist-test-slave-qgtk krb5kdc[11377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:06:00 dist-test-slave-qgtk krb5kdc[11377](info): set up 2 sockets
Jun 20 23:06:00 dist-test-slave-qgtk krb5kdc[11377](info): commencing operation
krb5kdc: starting...
W20260620 23:06:01.706365 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:06:01 dist-test-slave-qgtk krb5kdc[11377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:06:01.833741 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:33193
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:01.916091 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:01.916312 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:01.916337 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:01.917542 11390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:01.917590 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:01.917604 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:01.917618 11390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:01.917629 11390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:01.919245 11390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:01.919531 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:01.919780 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:01.924682 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:01.924695 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:01.924691 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:01.924875 11390 server_base.cc:1034] running on GCE node
I20260620 23:06:01.925155 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:01.925464 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:01.926628 11390 hybrid_clock.cc:648] HybridClock initialized: now 1781996761926617 us; error 28 us; skew 500 ppm
Jun 20 23:06:01 dist-test-slave-qgtk krb5kdc[11377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996761, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:01.932605 11390 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:06:01.933128 11390 webserver.cc:460] Webserver started at http://127.21.66.126:45903/ using document root <none> and password file <none>
I20260620 23:06:01.933359 11390 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:01.933429 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:01.933583 11390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:01.934744 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "307b00dcc73146db83a6a43c262350ef"
format_stamp: "Formatted at 2026-06-20 23:06:01 on dist-test-slave-qgtk"
server_key: "edcb01ed9a020e8f8bf2f23ff129a311"
server_key_iv: "fa40b49b6b7a14b770a73739464ebe67"
server_key_version: "encryptionkey@0"
I20260620 23:06:01.935105 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "307b00dcc73146db83a6a43c262350ef"
format_stamp: "Formatted at 2026-06-20 23:06:01 on dist-test-slave-qgtk"
server_key: "edcb01ed9a020e8f8bf2f23ff129a311"
server_key_iv: "fa40b49b6b7a14b770a73739464ebe67"
server_key_version: "encryptionkey@0"
I20260620 23:06:01.936446 11390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 23:06:01.937199 11405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:01.937500 11390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:06:01.937604 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "307b00dcc73146db83a6a43c262350ef"
format_stamp: "Formatted at 2026-06-20 23:06:01 on dist-test-slave-qgtk"
server_key: "edcb01ed9a020e8f8bf2f23ff129a311"
server_key_iv: "fa40b49b6b7a14b770a73739464ebe67"
server_key_version: "encryptionkey@0"
I20260620 23:06:01.937686 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:01.941183 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:01.943329 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:01.943495 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:01.947762 11390 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:33193
I20260620 23:06:01.947803 11457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:33193 every 8 connection(s)
I20260620 23:06:01.948166 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:01.948678 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11390
I20260620 23:06:01.948814 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:06:01.949004 21769 external_mini_cluster.cc:1428] Setting key c7e12bc7b02824a5a1d8d815db03893b
I20260620 23:06:01.949381 11458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:01.952059 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.949578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36673 (local address 127.21.66.126:33193)
0620 23:06:01.949792 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:01.949803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:01.949838 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:06:01.950696 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:01.950706 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:01.950745 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:01.950925 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:01.951583 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:01.951587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:01.951587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:01.951592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:01.951642 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:01.951659 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:01.951735 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:01.951802 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 23:06:01.951845 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260620 23:06:01.953166 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:06:01.953343 11458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef: Bootstrap starting.
I20260620 23:06:01.954187 11458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:01.954571 11458 log.cc:824] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:01.955300 11458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef: No bootstrap required, opened a new log
I20260620 23:06:01.956197 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:01.954979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50129 (local address 127.21.66.126:33193)
0620 23:06:01.955040 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:01.955042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:01.955108 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:06:01.955191 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:01.955192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:01.955198 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:01.955254 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:01.955963 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:01.955964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:01.955964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:06:01.955969 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:01.955984 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:06:01.955988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:01.956044 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:01.956110 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 23:06:01.956125 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:06:01.957583 11458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307b00dcc73146db83a6a43c262350ef" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33193 } }
I20260620 23:06:01.957722 11458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:06:01.957755 11458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307b00dcc73146db83a6a43c262350ef, State: Initialized, Role: FOLLOWER
I20260620 23:06:01.957897 11458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307b00dcc73146db83a6a43c262350ef" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33193 } }
I20260620 23:06:01.957979 11458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:06:01.958035 11458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:06:01.958088 11458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:01.958666 11458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307b00dcc73146db83a6a43c262350ef" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33193 } }
I20260620 23:06:01.958788 11458 leader_election.cc:302] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307b00dcc73146db83a6a43c262350ef; no voters:
I20260620 23:06:01.958962 11458 leader_election.cc:288] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:06:01.958999 11468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:01.959182 11468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [term 1 LEADER]: Becoming Leader. State: Replica: 307b00dcc73146db83a6a43c262350ef, State: Running, Role: LEADER
I20260620 23:06:01.959218 11458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:01.959434 11468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307b00dcc73146db83a6a43c262350ef" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33193 } }
I20260620 23:06:01.959909 11469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "307b00dcc73146db83a6a43c262350ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307b00dcc73146db83a6a43c262350ef" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33193 } } }
I20260620 23:06:01.959916 11470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307b00dcc73146db83a6a43c262350ef. Latest consensus state: current_term: 1 leader_uuid: "307b00dcc73146db83a6a43c262350ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307b00dcc73146db83a6a43c262350ef" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 33193 } } }
I20260620 23:06:01.960101 11469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:01.960116 11470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:01.960565 11477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:01.961089 11477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:01.962570 11477 catalog_manager.cc:1347] Generated new cluster ID: 850772caaeb04822b8c5490ca4c01c60
I20260620 23:06:01.962621 11477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:01.976649 11477 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:01.976893 11477 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:01.982898 11477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307b00dcc73146db83a6a43c262350ef: Generated new TSK 0
I20260620 23:06:01.983125 11477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:06:01.989815 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11390
2026-06-20T23:06:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1427 ms)
[----------] 18 tests from AuthTokenIssuingTest (24860 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:06:02 dist-test-slave-qgtk krb5kdc[11489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:06:02 dist-test-slave-qgtk krb5kdc[11489](info): set up 2 sockets
Jun 20 23:06:02 dist-test-slave-qgtk krb5kdc[11489](info): commencing operation
krb5kdc: starting...
W20260620 23:06:03.046486 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:06:03.165062 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34269
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34269,127.21.66.125:44313,127.21.66.124:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:03.251477 11502 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:06:03.251698 11502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:03.251714 11502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:03.251735 11502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:03.252825 11502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:03.252866 11502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:03.252880 11502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:03.252892 11502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:03.252904 11502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:03.254459 11502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34269,127.21.66.125:44313,127.21.66.124:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:03.254699 11502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:03.254871 11502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:03.259961 11507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.260061 11502 server_base.cc:1034] running on GCE node
W20260620 23:06:03.260082 11510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:03.259961 11508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.260511 11502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:03.260783 11502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:03.261961 11502 hybrid_clock.cc:648] HybridClock initialized: now 1781996763261926 us; error 48 us; skew 500 ppm
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:03.268843 11502 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:06:03.269375 11502 webserver.cc:460] Webserver started at http://127.21.66.126:46749/ using document root <none> and password file <none>
I20260620 23:06:03.269616 11502 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:03.269690 11502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:03.269845 11502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:03.271076 11502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "3c9d60c9d9014a3795c6456689e570d0"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "94c4e0934bf98c5d43e0d949ec8fc73d"
server_key_iv: "65cc3abcb7c2384d0b1184b36c887e89"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.271471 11502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "3c9d60c9d9014a3795c6456689e570d0"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "94c4e0934bf98c5d43e0d949ec8fc73d"
server_key_iv: "65cc3abcb7c2384d0b1184b36c887e89"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.272903 11502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 23:06:03.273763 11517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.273958 11502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:06:03.274070 11502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "3c9d60c9d9014a3795c6456689e570d0"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "94c4e0934bf98c5d43e0d949ec8fc73d"
server_key_iv: "65cc3abcb7c2384d0b1184b36c887e89"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.274255 11502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:03.290683 11502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:03.293022 11502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:03.293249 11502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:03.297465 11502 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:34269
I20260620 23:06:03.297498 11569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:34269 every 8 connection(s)
I20260620 23:06:03.297860 11502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:03.298893 11570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:03.300201 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11502
I20260620 23:06:03.300346 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:06:03.300531 21769 external_mini_cluster.cc:1428] Setting key beeecab961d3a67769caf363c6a5ed17
I20260620 23:06:03.300695 11570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.125@KRBTEST.COM" created.
I20260620 23:06:03.310940 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.301449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34269 (local address 127.0.0.1:44782)
0620 23:06:03.301638 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.301684 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.302247 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.302545 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.302547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.302863 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.303022 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.303027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.304134 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.304135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.304813 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.304816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.305347 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.305348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.305412 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.305988 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.305998 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.307464 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.309325 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.309332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.309335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.309539 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.309787 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.309788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.309788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.309883 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.310313 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.310318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.310380 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.310498 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.310735 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":94,"thread_start_us":86,"threads_started":1}
I20260620 23:06:03.310940 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.301545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.21.66.126:34269)
0620 23:06:03.301721 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.301726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.301765 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:06:03.302406 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.302409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.302435 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.302538 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.303134 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.304018 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.304915 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.305232 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.305362 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.307586 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.307595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.307596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.307604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.309220 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.309620 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.309624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.309624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.309716 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.309948 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.309949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.309949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.310244 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.310335 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.310528 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.310678 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260620 23:06:03.311681 11570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:03.312067 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.311939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53780)
0620 23:06:03.311988 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.312035 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:06:03.312211 11518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
I20260620 23:06:03.312630 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.312540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53794)
0620 23:06:03.312587 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.312610 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:06:03.312736 11570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:44313: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab.
Entry for principal kudu/127.21.66.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab.
I20260620 23:06:03.325618 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:44313
--webserver_interface=127.21.66.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab
--principal=kudu/127.21.66.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34269,127.21.66.125:44313,127.21.66.124:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:06:03.362103 11570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } attempt: 1
I20260620 23:06:03.362852 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.362671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53796)
0620 23:06:03.362762 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.362818 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 23:06:03.363306 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.363194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53798)
0620 23:06:03.363249 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.363277 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:06:03.363428 11570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:44313: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
W20260620 23:06:03.417970 11577 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:06:03.418267 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:03.418289 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:03.418314 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:03.419798 11577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:03.419852 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:03.419867 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:03.419883 11577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:03.419895 11577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:03.421782 11577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34269,127.21.66.125:44313,127.21.66.124:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab
--principal=kudu/127.21.66.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:44313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.66.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:03.422091 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:03.422389 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:03.428144 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:03.428182 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.428237 11577 server_base.cc:1034] running on GCE node
W20260620 23:06:03.428156 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.428804 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:03.429158 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:03.430332 11577 hybrid_clock.cc:648] HybridClock initialized: now 1781996763430319 us; error 46 us; skew 500 ppm
I20260620 23:06:03.431578 11570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } attempt: 2
I20260620 23:06:03.432039 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.431853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53808)
0620 23:06:03.431932 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.431973 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 23:06:03.432479 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.432369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53816)
0620 23:06:03.432426 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.432455 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 23:06:03.432603 11570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:44313: Network error: Client connection negotiation failed: client connection to 127.21.66.125:44313: connect: Connection refused (error 111)
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:03.433509 11577 init.cc:377] Logged in from keytab as kudu/127.21.66.125@KRBTEST.COM (short username kudu)
I20260620 23:06:03.434001 11577 webserver.cc:460] Webserver started at http://127.21.66.125:43679/ using document root <none> and password file <none>
I20260620 23:06:03.434266 11577 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:03.434314 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:03.434473 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:03.435689 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data/instance:
uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "d2d909a3028b80d292c93737d31cf13b"
server_key_iv: "4887969abe4d17a715759671fffc6634"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.436076 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/wal/instance:
uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "d2d909a3028b80d292c93737d31cf13b"
server_key_iv: "4887969abe4d17a715759671fffc6634"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.437481 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 23:06:03.438436 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.438678 11577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:06:03.438782 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/wal
uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "d2d909a3028b80d292c93737d31cf13b"
server_key_iv: "4887969abe4d17a715759671fffc6634"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.438866 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:03.449156 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:03.451639 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:03.451817 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:03.457264 11577 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.125:44313
I20260620 23:06:03.457302 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.125:44313 every 8 connection(s)
I20260620 23:06:03.458000 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
I20260620 23:06:03.459419 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:03.461464 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11577
I20260620 23:06:03.461594 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-1/wal/instance
I20260620 23:06:03.461794 21769 external_mini_cluster.cc:1428] Setting key f8f3238928a1aaf8b8e31d1df936db11
I20260620 23:06:03.462145 11645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.124@KRBTEST.COM" created.
I20260620 23:06:03.475411 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.463143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34269 (local address 127.0.0.1:44796)
0620 23:06:03.463723 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.463830 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.464383 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.464929 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.464933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.465386 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.465629 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.465635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.466724 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.466725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.468045 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.468048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.468497 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.468498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.468637 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.469391 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.469403 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.471320 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.473470 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.473481 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.473483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.473883 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.474172 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.474173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.474173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.474323 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.474668 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.474676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.474846 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.474961 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.475166 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":314,"threads_started":1}
I20260620 23:06:03.476042 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.463716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.21.66.126:34269)
0620 23:06:03.463769 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.463771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.463933 (+ 162us) server_negotiation.cc:407] Connection header received
0620 23:06:03.464504 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.464506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.464522 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.464570 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.465737 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.466605 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.468157 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.468386 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.468476 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.471449 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.471451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.471452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.471462 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.473389 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.473993 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.473995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.473995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.474072 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.474409 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.474410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.474411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.474575 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.474654 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.475903 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.475980 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:06:03.476531 11645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:06:03.486881 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.476852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53832)
0620 23:06:03.476954 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.476965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.477005 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.477361 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.477362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.477558 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.477629 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.477632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.479288 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.479289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.480160 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.480163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.480850 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.480852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.480941 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.481465 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.481471 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.483048 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.485601 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.485602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.485602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.485808 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.486056 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.486057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.486058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.486176 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.486514 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.486515 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.486557 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.486622 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.486728 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:06:03.487214 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.477042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.21.66.125:44313)
0620 23:06:03.477200 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.477205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.477231 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:06:03.477253 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.477255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.477269 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.477704 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.477746 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.479176 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.480261 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.480770 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.480848 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.483124 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.483134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.483135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.483143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.485509 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.485903 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.485907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.485908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.485972 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.486250 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.486250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.486251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.486435 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.486536 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.487050 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.487146 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260620 23:06:03.487875 11645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:03.488361 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.488197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38006)
0620 23:06:03.488271 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.488329 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 23:06:03.488525 11593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
I20260620 23:06:03.489094 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.488997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38014)
0620 23:06:03.489041 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.489067 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 23:06:03.489238 11645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:34103: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab.
Entry for principal kudu/127.21.66.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab.
I20260620 23:06:03.492628 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:34103
--webserver_interface=127.21.66.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab
--principal=kudu/127.21.66.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:34269,127.21.66.125:44313,127.21.66.124:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:06:03.523792 11570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } attempt: 3
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:06:03.531659 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.524343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53838)
0620 23:06:03.524414 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.524430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.524503 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.524732 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.524733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.524903 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.524968 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.524970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.525651 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.525652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.526301 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.526304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.526720 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.526721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.526808 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.527291 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.527301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.528805 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.530577 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.530579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.530579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.530770 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.530980 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.530981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.530981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.531047 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.531301 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.531302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.531348 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.531435 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.531541 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 23:06:03.531649 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.524442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.21.66.125:44313)
0620 23:06:03.524517 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.524519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.524531 (+ 12us) server_negotiation.cc:407] Connection header received
0620 23:06:03.524563 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.524564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.524577 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.524710 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.525066 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.525563 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.526386 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.526635 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.526723 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.528905 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.528906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.528906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.528914 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.530461 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.530850 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.530851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.530851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.530911 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.531110 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.531111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.531112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.531233 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.531309 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.531460 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.531544 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:06:03.532003 11570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:03.532373 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.532271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38022)
0620 23:06:03.532322 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.532349 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 23:06:03.532760 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.532702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38028)
0620 23:06:03.532728 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.532748 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 23:06:03.532859 11570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:34103: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
I20260620 23:06:03.538484 11645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } attempt: 1
I20260620 23:06:03.538981 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.538838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38032)
0620 23:06:03.538903 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.538950 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 23:06:03.539382 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.539283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38036)
0620 23:06:03.539329 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.539360 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 23:06:03.539486 11645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:34103: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
I20260620 23:06:03.564005 11570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } attempt: 1
I20260620 23:06:03.564479 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.564348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38040)
0620 23:06:03.564408 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.564449 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 23:06:03.564785 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.564694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38042)
0620 23:06:03.564739 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.564769 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:06:03.564875 11570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:34103: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
W20260620 23:06:03.585409 11652 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:06:03.585644 11652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:03.585664 11652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:03.585721 11652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:03.587085 11652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:03.587128 11652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:03.587143 11652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:03.587158 11652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:03.587211 11652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:03.588902 11652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:34269,127.21.66.125:44313,127.21.66.124:34103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab
--principal=kudu/127.21.66.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.66.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:03.589182 11652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:03.589433 11652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:03.594839 11660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:03.594831 11657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.594867 11652 server_base.cc:1034] running on GCE node
W20260620 23:06:03.594827 11658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:03.595233 11652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:03.595533 11652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:03.596730 11652 hybrid_clock.cc:648] HybridClock initialized: now 1781996763596714 us; error 26 us; skew 500 ppm
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:03.599402 11652 init.cc:377] Logged in from keytab as kudu/127.21.66.124@KRBTEST.COM (short username kudu)
I20260620 23:06:03.599989 11652 webserver.cc:460] Webserver started at http://127.21.66.124:37915/ using document root <none> and password file <none>
I20260620 23:06:03.600260 11652 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:03.600333 11652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:03.600517 11652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:03.601724 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data/instance:
uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "b898d1dded0ffc7c394ab26543e007d9"
server_key_iv: "0412bfe4de59b1e408a41bff0ddb46bf"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.602167 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/wal/instance:
uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "b898d1dded0ffc7c394ab26543e007d9"
server_key_iv: "0412bfe4de59b1e408a41bff0ddb46bf"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.603632 11652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 23:06:03.604419 11667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:03.604593 11652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 23:06:03.604697 11652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/wal
uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce"
format_stamp: "Formatted at 2026-06-20 23:06:03 on dist-test-slave-qgtk"
server_key: "b898d1dded0ffc7c394ab26543e007d9"
server_key_iv: "0412bfe4de59b1e408a41bff0ddb46bf"
server_key_version: "encryptionkey@0"
I20260620 23:06:03.604797 11652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:03.607640 11645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } attempt: 2
I20260620 23:06:03.608193 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.608039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38054)
0620 23:06:03.608119 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.608163 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:06:03.608600 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.608496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38070)
0620 23:06:03.608528 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.608556 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 23:06:03.608759 11645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:34103: Network error: Client connection negotiation failed: client connection to 127.21.66.124:34103: connect: Connection refused (error 111)
I20260620 23:06:03.618732 11652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:03.620739 11652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:03.620852 11652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:03.625238 11652 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.124:34103
I20260620 23:06:03.625308 11719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.124:34103 every 8 connection(s)
I20260620 23:06:03.625672 11652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
I20260620 23:06:03.626853 11720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:03.628576 11720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:03.628883 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11652
I20260620 23:06:03.628986 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781996487602640-21769-0/minicluster-data/master-2/wal/instance
I20260620 23:06:03.629190 21769 external_mini_cluster.cc:1428] Setting key 92b2fbf7c725d6561360984f69ca2df3
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
I20260620 23:06:03.639742 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.629423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.21.66.126:34269)
0620 23:06:03.629511 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.629514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.629745 (+ 231us) server_negotiation.cc:407] Connection header received
0620 23:06:03.630341 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.630343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.630355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.630419 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.631172 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.632043 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.633251 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.633541 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.633626 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.636254 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.636256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.636256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.636262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.638252 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.638658 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.638659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.638660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.638722 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.638989 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.638990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.638990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.639104 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.639156 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.639510 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.639594 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:06:03.639878 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.629414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:34269 (local address 127.0.0.1:44806)
0620 23:06:03.629605 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.629661 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.630244 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.630455 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.630457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.630832 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.631068 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.631074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.632133 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.632134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.633148 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.633151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.633651 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.633652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.633750 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.634419 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.634430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.636146 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.638332 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.638338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.638340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.638569 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.638794 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.638795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.638795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.638892 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.639172 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.639179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.639348 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.639464 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.639638 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260620 23:06:03.640172 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.629587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.21.66.124:34103)
0620 23:06:03.629701 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.629706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.629898 (+ 192us) server_negotiation.cc:407] Connection header received
0620 23:06:03.630264 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.630272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.630311 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.630427 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.631181 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.632418 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.633410 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.633867 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.633972 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.636870 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.636879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.636879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.636887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.638878 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.639254 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.639256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.639257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.639306 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.639504 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.639505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.639505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.639687 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.639840 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.639925 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.639996 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
I20260620 23:06:03.640108 11570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } attempt: 2
I20260620 23:06:03.641003 11720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:06:03.649171 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.641329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.21.66.125:44313)
0620 23:06:03.641382 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.641384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.641391 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:06:03.641414 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.641415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.641425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.641484 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.641796 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.642505 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.643386 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.643700 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.643800 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.646030 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.646031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.646031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.646038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.647713 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.648288 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.648290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.648291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.648348 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.648568 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.648569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.648569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.648719 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.648786 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.648956 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.649039 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 23:06:03.649442 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.640586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38096)
0620 23:06:03.640712 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.640722 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.640759 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.640901 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.640902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.641072 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.641130 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.641133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.642018 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.642019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.642734 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.642736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.643189 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.643190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.643245 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.643671 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.643676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.645169 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.648181 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.648182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.648183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.648582 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.648783 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.648784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.648785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.648844 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.649118 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.649119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.649178 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.649245 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.649345 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 23:06:03.649539 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.641258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:44313 (local address 127.0.0.1:53844)
0620 23:06:03.641296 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.641305 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.641333 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.641499 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.641500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.641648 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.641692 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.641694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.642597 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.642598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.643310 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.643312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.643802 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.643802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.643873 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.644844 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.644849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.645950 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.647817 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.647818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.647818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.648141 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.648425 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.648426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.648426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.648497 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.648776 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.648778 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.648825 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.649329 (+ 504us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.649425 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 23:06:03.649829 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.640722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.21.66.124:34103)
0620 23:06:03.640759 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.640761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.640769 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:06:03.640813 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.640814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.640824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.641199 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.641230 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.641934 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.642815 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.643116 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.643181 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.646365 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.646366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.646366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.646373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.648083 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.648653 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.648654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.648654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.648701 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.648920 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.648921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.648921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.649033 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.649120 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.649639 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.649721 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:06:03.650025 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.641014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.21.66.126:34269)
0620 23:06:03.641046 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.641048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.641056 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:06:03.641127 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.641127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.641137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.641182 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.641760 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.642530 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.643416 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.643681 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.643742 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.646865 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.646866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.646867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.646873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.648671 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.649072 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.649073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.649073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.649127 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.649335 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.649336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.649336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.649459 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.649644 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.649868 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.649941 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:06:03.650074 11720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:03.651793 11570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0: Bootstrap starting.
I20260620 23:06:03.652576 11570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:03.652894 11570 log.cc:824] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:03.654049 11570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0: No bootstrap required, opened a new log
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:06:03.656481 11570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.656647 11570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9d60c9d9014a3795c6456689e570d0, State: Initialized, Role: FOLLOWER
I20260620 23:06:03.656749 11570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.657083 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.657246 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:03.657152 11570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:03.657939 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.650372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38108)
0620 23:06:03.650443 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.650452 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.650491 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.650640 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.650641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.650799 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.650838 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.650841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.651711 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.651713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.652292 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.652294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.652742 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.652743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.652840 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.653295 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.653300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.654624 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.656279 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.656279 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.656280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.656500 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.657153 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.657153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.657154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.657238 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.657567 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.657568 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.657608 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.657680 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.657784 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 23:06:03.657950 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.650442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.21.66.124:34103)
0620 23:06:03.650476 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.650478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.650486 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:06:03.650546 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.650547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.650555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.650607 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.650913 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.651637 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.652386 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.652581 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.652699 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.654752 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.654753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.654754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.654760 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.656187 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.657020 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.657021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.657021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.657076 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.657304 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.657305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.657305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.657498 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.657584 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.657702 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.657830 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":19}
I20260620 23:06:03.659212 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.650848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.21.66.125:44313)
0620 23:06:03.650882 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.650884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.650892 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:06:03.650926 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.650927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.650935 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.650983 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.651540 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.652175 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.653106 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.653404 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.653484 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.656163 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.656165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.656165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.656173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.658023 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.658430 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.658431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.658431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.658488 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.658690 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.658691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.658691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.658823 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.658992 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.659027 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.659115 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 23:06:03.659715 11720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: Bootstrap starting.
I20260620 23:06:03.660744 11720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:03.660950 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:06:03.661068 11720 log.cc:824] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:03.661891 11720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: No bootstrap required, opened a new log
I20260620 23:06:03.661948 11544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3c9d60c9d9014a3795c6456689e570d0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44812
I20260620 23:06:03.662068 11544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:06:03.662144 11544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:03.662827 11544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.663096 11544 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [CANDIDATE]: Term 1 election: Requested vote from peers d28d5b6dcf324f2da05fd8c9ea1327a1 (127.21.66.125:44313), b7f472efd7cc4a3c9bc76baceb7bc9ce (127.21.66.124:34103)
I20260620 23:06:03.660792 11502 master_runner.cc:186] Error getting master registration for 127.21.66.125:44313: OK, instance_id { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" instance_seqno: 1781996763455521 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:06:03.663820 11502 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:06:03.664127 11720 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.664322 11720 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f472efd7cc4a3c9bc76baceb7bc9ce, State: Initialized, Role: FOLLOWER
I20260620 23:06:03.664431 11720 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.664707 11745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.664813 11694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c9d60c9d9014a3795c6456689e570d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce"
I20260620 23:06:03.664911 11694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:03.665709 11694 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9d60c9d9014a3795c6456689e570d0 in term 1.
W20260620 23:06:03.665721 11737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:06:03.665822 11737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:06:03.665930 11518 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9d60c9d9014a3795c6456689e570d0, b7f472efd7cc4a3c9bc76baceb7bc9ce; no voters:
I20260620 23:06:03.664868 11745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:03.664769 11720 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:03.666774 11726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:03.666949 11726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9d60c9d9014a3795c6456689e570d0, State: Running, Role: LEADER
I20260620 23:06:03.667055 11726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.667541 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9d60c9d9014a3795c6456689e570d0. Latest consensus state: current_term: 1 leader_uuid: "3c9d60c9d9014a3795c6456689e570d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.667809 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:03.668318 11761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:03.668704 11761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:03.669277 11652 master_runner.cc:186] Error getting master registration for 127.21.66.125:44313: OK, instance_id { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" instance_seqno: 1781996763455521 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:06:03.672912 11652 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:06:03.675230 11759 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:06:03.675318 11759 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:06:03.675989 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.664350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.21.66.126:34269)
0620 23:06:03.664394 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.664397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.664684 (+ 287us) server_negotiation.cc:407] Connection header received
0620 23:06:03.667519 (+ 2835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.667521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.667533 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.667566 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.668932 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.669717 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.670751 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.671037 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.671091 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.672574 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.672575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.672576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.672584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.674250 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.674835 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.674836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.674837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.674906 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.675214 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.675215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.675216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.675362 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.675432 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.675817 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.675896 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:06:03.676463 11694 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [term 1 FOLLOWER]: Refusing update from remote peer 3c9d60c9d9014a3795c6456689e570d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:06:03.676543 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.664379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.21.66.125:44313)
0620 23:06:03.664423 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.664425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.664893 (+ 468us) server_negotiation.cc:407] Connection header received
0620 23:06:03.664993 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.664994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.665004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.665036 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.665876 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.666694 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.668598 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.668897 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.668947 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.671551 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.671553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.671553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.671561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.673838 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.674356 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.674358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.674359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.674421 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.674639 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.674640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.674640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.674777 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.674846 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.676358 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.676447 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 23:06:03.677242 11753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:06:03.678696 11745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9d60c9d9014a3795c6456689e570d0. Latest consensus state: current_term: 1 leader_uuid: "3c9d60c9d9014a3795c6456689e570d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.678902 11745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:03.681613 11753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c9d60c9d9014a3795c6456689e570d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.681712 11753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:03.682317 11745 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c9d60c9d9014a3795c6456689e570d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.682418 11745 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:03.682538 11762 mvcc.cc:205] Tried to move back new op lower bound from 7299058744015454208 to 7299058743981133824. Current Snapshot: MvccSnapshot[applied={T|T < 7299058744015454208}]
I20260620 23:06:03.683233 11764 mvcc.cc:205] Tried to move back new op lower bound from 7299058744015454208 to 7299058743981133824. Current Snapshot: MvccSnapshot[applied={T|T < 7299058744015454208}]
I20260620 23:06:03.683444 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.664738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.21.66.124:34103)
0620 23:06:03.664774 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.664776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.670696 (+ 5920us) server_negotiation.cc:407] Connection header received
0620 23:06:03.670788 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.670789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.670798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.670827 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.674912 (+ 4085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.675623 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.678510 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.678798 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.678844 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.680485 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.680486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.680487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.680493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.682187 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.682578 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.682579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.682580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.682638 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.682839 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.682840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.682840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.682964 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.683052 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.683261 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.683340 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:06:03.684325 11761 catalog_manager.cc:1347] Generated new cluster ID: 1e50b2cdea714bbe9ffa0134f85654cb
I20260620 23:06:03.684377 11761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:03.694535 11761 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:03.695273 11761 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:03.699898 11645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } attempt: 3
I20260620 23:06:03.703492 11761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0: Generated new TSK 0
I20260620 23:06:03.703795 11761 catalog_manager.cc:1514] Initializing in-progress tserver states...
Jun 20 23:06:03 dist-test-slave-qgtk krb5kdc[11489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996763, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
I20260620 23:06:03.709194 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.700378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.21.66.124:34103)
0620 23:06:03.700425 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.700427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.700434 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:06:03.700455 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.700456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.700465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.700509 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.700886 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.701655 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.702692 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.703031 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.703091 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.706099 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:03.706100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:03.706101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:03.706108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:03.707790 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.708227 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.708228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.708228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.708282 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:03.708462 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:03.708462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:03.708463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:03.708588 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:03.708641 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:03.709019 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 23:06:03.709090 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 23:06:03.709323 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:03.700177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:34103 (local address 127.0.0.1:38120)
0620 23:06:03.700259 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:03.700271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:03.700308 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:03.700551 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:03.700552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:03.700754 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:03.700803 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.700805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.701762 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.701763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.702584 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:03.702587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:03.703881 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:03.703882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:03.703945 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:03.704500 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:03.704505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:03.706007 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:03.707918 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.707919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.707920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.708134 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.708347 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:03.708348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:03.708348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:03.708402 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:03.708665 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:03.708666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:03.708708 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:06:03.708767 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 23:06:03.708862 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:06:03.710963 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: Bootstrap starting.
I20260620 23:06:03.711747 11645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:03.712067 11645 log.cc:824] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:03.719466 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: No bootstrap required, opened a new log
I20260620 23:06:03.721539 11645 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.721699 11645 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28d5b6dcf324f2da05fd8c9ea1327a1, State: Initialized, Role: FOLLOWER
I20260620 23:06:03.721781 11645 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } }
I20260620 23:06:03.722105 11645 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:03.722204 11619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:03.723137 11619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [term 1 FOLLOWER]: Refusing update from remote peer 3c9d60c9d9014a3795c6456689e570d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:06:03.723331 11617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c9d60c9d9014a3795c6456689e570d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1"
I20260620 23:06:03.724126 11617 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c9d60c9d9014a3795c6456689e570d0 in term 1.
I20260620 23:06:03.724709 11726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:06:03.726377 11781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:06:03.726430 11781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:06:03.722101 11770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.729331 11770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:03.729427 11770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9d60c9d9014a3795c6456689e570d0. Latest consensus state: current_term: 1 leader_uuid: "3c9d60c9d9014a3795c6456689e570d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.729538 11770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:03.729614 11770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c9d60c9d9014a3795c6456689e570d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:03.729712 11770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:03.731192 11785 mvcc.cc:205] Tried to move back new op lower bound from 7299058744124604416 to 7299058743981133824. Current Snapshot: MvccSnapshot[applied={T|T < 7299058744015454208}]
W20260620 23:06:03.748310 11573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.746714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.21.66.126:34269)
0620 23:06:03.746755 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.746757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.747537 (+ 780us) server_negotiation.cc:407] Connection header received
0620 23:06:03.747660 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.747661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.747673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.747706 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.748227 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44820: BlockingRecv error: recv got EOF from 127.0.0.1:44820 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 23:06:03.749097 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.746470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.21.66.125:44313)
0620 23:06:03.746520 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.746523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.748533 (+ 2010us) server_negotiation.cc:407] Connection header received
0620 23:06:03.748632 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.748646 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.748657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.748687 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.749035 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53868: BlockingRecv error: recv got EOF from 127.0.0.1:53868 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 23:06:03.749847 11723 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.747269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.21.66.124:34103)
0620 23:06:03.747311 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.747313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.749342 (+ 2029us) server_negotiation.cc:407] Connection header received
0620 23:06:03.749438 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.749439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.749450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.749478 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.749782 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38126: BlockingRecv error: recv got EOF from 127.0.0.1:38126 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:06:03.752436 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.751711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.21.66.125:44313)
0620 23:06:03.751745 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.751747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.751928 (+ 181us) server_negotiation.cc:407] Connection header received
0620 23:06:03.752036 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.752037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.752056 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.752085 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.752387 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53878: BlockingRecv error: recv got EOF from 127.0.0.1:53878 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 23:06:03.755491 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.754781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.21.66.125:44313)
0620 23:06:03.754814 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.754815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.755020 (+ 205us) server_negotiation.cc:407] Connection header received
0620 23:06:03.755111 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.755111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.755120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.755149 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.755445 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53890: BlockingRecv error: recv got EOF from 127.0.0.1:53890 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 23:06:03.759459 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.758741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.21.66.125:44313)
0620 23:06:03.758775 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.758777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.758975 (+ 198us) server_negotiation.cc:407] Connection header received
0620 23:06:03.759067 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.759069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.759078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.759106 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.759410 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53906: BlockingRecv error: recv got EOF from 127.0.0.1:53906 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 23:06:03.765398 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.764649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.21.66.125:44313)
0620 23:06:03.764690 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.764692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.764899 (+ 207us) server_negotiation.cc:407] Connection header received
0620 23:06:03.764995 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.764996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.765007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.765036 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.765347 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53918: BlockingRecv error: recv got EOF from 127.0.0.1:53918 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 23:06:03.775717 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.774849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.21.66.125:44313)
0620 23:06:03.774894 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.774896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.775138 (+ 242us) server_negotiation.cc:407] Connection header received
0620 23:06:03.775242 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.775243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.775254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.775285 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.775656 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53934: BlockingRecv error: recv got EOF from 127.0.0.1:53934 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 23:06:03.793936 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.793065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.21.66.125:44313)
0620 23:06:03.793110 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.793112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.793347 (+ 235us) server_negotiation.cc:407] Connection header received
0620 23:06:03.793457 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.793458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.793470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.793500 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.793875 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53940: BlockingRecv error: recv got EOF from 127.0.0.1:53940 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:06:03.828593 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.827666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.21.66.125:44313)
0620 23:06:03.827715 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.827718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.827986 (+ 268us) server_negotiation.cc:407] Connection header received
0620 23:06:03.828094 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.828095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.828107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.828140 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.828530 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53946: BlockingRecv error: recv got EOF from 127.0.0.1:53946 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 23:06:03.894951 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:03.894235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.21.66.125:44313)
0620 23:06:03.894309 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:03.894311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:03.894327 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:06:03.894413 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:03.894414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:03.894425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:03.894538 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:03.894879 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53954: BlockingRecv error: recv got EOF from 127.0.0.1:53954 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 23:06:04.025274 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:04.024301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.21.66.125:44313)
0620 23:06:04.024367 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.024370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.024487 (+ 117us) server_negotiation.cc:407] Connection header received
0620 23:06:04.024605 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.024607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.024620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.024704 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:04.025188 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53960: BlockingRecv error: recv got EOF from 127.0.0.1:53960 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 23:06:04.279610 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c9d60c9d9014a3795c6456689e570d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9d60c9d9014a3795c6456689e570d0" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 34269 } } peers { permanent_uuid: "d28d5b6dcf324f2da05fd8c9ea1327a1" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 44313 } } peers { permanent_uuid: "b7f472efd7cc4a3c9bc76baceb7bc9ce" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 34103 } } }
I20260620 23:06:04.279793 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9d60c9d9014a3795c6456689e570d0 [sys.catalog]: This master's current role is: LEADER
W20260620 23:06:04.283274 11650 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:04.282530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.21.66.125:44313)
0620 23:06:04.282598 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.282602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.282646 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:06:04.282739 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.282740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.282754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.282823 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:04.283189 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53962: BlockingRecv error: recv got EOF from 127.0.0.1:53962 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 23:06:04.675782 11759 catalog_manager.cc:1259] Loaded cluster ID: 1e50b2cdea714bbe9ffa0134f85654cb
I20260620 23:06:04.675865 11759 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: loading cluster ID for follower catalog manager: success
I20260620 23:06:04.676765 11759 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: acquiring CA information for follower catalog manager: success
I20260620 23:06:04.677136 11759 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7f472efd7cc4a3c9bc76baceb7bc9ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:06:04.726861 11781 catalog_manager.cc:1259] Loaded cluster ID: 1e50b2cdea714bbe9ffa0134f85654cb
I20260620 23:06:04.726938 11781 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: loading cluster ID for follower catalog manager: success
I20260620 23:06:04.727789 11781 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: acquiring CA information for follower catalog manager: success
I20260620 23:06:04.728118 11781 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d28d5b6dcf324f2da05fd8c9ea1327a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:06:04.799070 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.796407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.21.66.125:44313)
0620 23:06:04.796510 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.796512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.796522 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:06:04.796609 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.796610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.796672 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.796760 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:04.796967 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.797454 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.798311 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.798511 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.798587 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.798735 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:04.798845 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:04.798905 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:04.798981 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 23:06:04.798999 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260620 23:06:04.803537 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.801069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.21.66.124:34103)
0620 23:06:04.801194 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.801196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.801205 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:06:04.801303 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.801303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.801345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.801412 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:04.801506 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.801965 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.802818 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.803011 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.803113 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.803258 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:04.803354 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:04.803406 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:04.803449 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 23:06:04.803466 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260620 23:06:04.807755 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.805181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.21.66.125:44313)
0620 23:06:04.805228 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.805230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.805322 (+ 92us) server_negotiation.cc:407] Connection header received
0620 23:06:04.805435 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.805437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.805451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.805516 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:04.805676 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.806215 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.807019 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.807296 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.807364 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.807517 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:04.807549 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:04.807603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:04.807655 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:06:04.807676 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 23:06:04.808425 11868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.805163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.21.66.126:34269)
0620 23:06:04.805257 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.805259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.805372 (+ 113us) server_negotiation.cc:407] Connection header received
0620 23:06:04.805425 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.805426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.805474 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.805574 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:04.805719 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.806336 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.807414 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.807736 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.807798 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.808031 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:04.808164 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:04.808225 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:04.808296 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 23:06:04.808323 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":40,"threads_started":1}
I20260620 23:06:04.809406 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:04.805411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.21.66.124:34103)
0620 23:06:04.805568 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:04.805569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:04.805577 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:06:04.805601 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:04.805601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:04.805609 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:04.805681 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:04.806345 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.807114 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.808420 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:04.808736 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:04.808798 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:04.809115 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:04.809150 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:04.809215 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:04.809307 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:06:04.809331 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 23:06:04.810257 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11502
I20260620 23:06:04.815613 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11577
I20260620 23:06:04.820734 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11652
2026-06-20T23:06:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2832 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:06:04 dist-test-slave-qgtk krb5kdc[11872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:06:04 dist-test-slave-qgtk krb5kdc[11872](info): set up 2 sockets
Jun 20 23:06:04 dist-test-slave-qgtk krb5kdc[11872](info): commencing operation
krb5kdc: starting...
W20260620 23:06:05.858709 21769 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/test-user.keytab.
Jun 20 23:06:05 dist-test-slave-qgtk krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.66.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
Entry for principal kudu/127.21.66.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab.
I20260620 23:06:05.989604 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39913
--webserver_interface=127.21.66.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39913,127.21.66.125:33051,127.21.66.124:36059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:06:06.076272 11885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:06:06.076506 11885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:06.076524 11885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:06.076546 11885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:06.077760 11885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:06.077808 11885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:06.077822 11885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:06.077836 11885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:06.077847 11885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:06.079421 11885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39913,127.21.66.125:33051,127.21.66.124:36059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.126.keytab
--principal=kudu/127.21.66.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.126:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.66.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:06.079727 11885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:06.079980 11885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:06.085237 11893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:06.085229 11890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:06.085232 11891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:06.085516 11885 server_base.cc:1034] running on GCE node
I20260620 23:06:06.085749 11885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:06.086083 11885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:06.087301 11885 hybrid_clock.cc:648] HybridClock initialized: now 1781996766087273 us; error 45 us; skew 500 ppm
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:06.090097 11885 init.cc:377] Logged in from keytab as kudu/127.21.66.126@KRBTEST.COM (short username kudu)
I20260620 23:06:06.090690 11885 webserver.cc:460] Webserver started at http://127.21.66.126:39333/ using document root <none> and password file <none>
I20260620 23:06:06.090935 11885 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:06.091008 11885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:06.091171 11885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:06.092291 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data/instance:
uuid: "04f339ea0f0047ed90d4a8c013104b38"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "eec7330af82384a5df45454d65f96b11"
server_key_iv: "dec4cbee1b582518b4ad9589f161cffa"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.092694 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/wal/instance:
uuid: "04f339ea0f0047ed90d4a8c013104b38"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "eec7330af82384a5df45454d65f96b11"
server_key_iv: "dec4cbee1b582518b4ad9589f161cffa"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.094281 11885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:06:06.095197 11900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:06.095387 11885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:06:06.095494 11885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/wal
uuid: "04f339ea0f0047ed90d4a8c013104b38"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "eec7330af82384a5df45454d65f96b11"
server_key_iv: "dec4cbee1b582518b4ad9589f161cffa"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.095571 11885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:06.105314 11885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:06.107640 11885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:06.107818 11885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:06.112344 11885 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.126:39913
I20260620 23:06:06.112399 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.126:39913 every 8 connection(s)
I20260620 23:06:06.112756 11885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/data/info.pb
I20260620 23:06:06.113794 11953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:06.114928 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11885
I20260620 23:06:06.115042 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-0/wal/instance
I20260620 23:06:06.115240 21769 external_mini_cluster.cc:1428] Setting key c4ed1920d209ae8ff56f6f674fd3413b
I20260620 23:06:06.115953 11953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.66.125@KRBTEST.COM; defaulting to no policy
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.125@KRBTEST.COM" created.
I20260620 23:06:06.126788 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.116857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39913 (local address 127.0.0.1:33818)
0620 23:06:06.117090 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.117173 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.117687 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.117981 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.117984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.118331 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.118488 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.118493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.119572 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.119574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.120299 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.120301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.120820 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.120821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.120886 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.121501 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.121510 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.123121 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.125110 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.125117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.125119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.125373 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.125661 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.125662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.125662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.125778 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.126275 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.126280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.126340 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.126413 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.126541 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260620 23:06:06.126819 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.117069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.21.66.126:39913)
0620 23:06:06.117200 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.117205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.117287 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:06:06.117854 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.117855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.117879 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.117976 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.118587 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.119453 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.120398 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.120709 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.120837 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.123204 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.123210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.123211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.123217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.124976 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.125486 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.125490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.125490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.125584 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.125854 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.125855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.125856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.126194 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.126281 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.126489 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.126634 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":106,"thread_start_us":54,"threads_started":1}
I20260620 23:06:06.127640 11953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:06.128018 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.127904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:42990)
0620 23:06:06.127952 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.128001 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 23:06:06.128186 11901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
I20260620 23:06:06.128651 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.128539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:42992)
0620 23:06:06.128592 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.128623 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 23:06:06.128772 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:33051: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab.
Entry for principal kudu/127.21.66.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab.
I20260620 23:06:06.141873 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:33051
--webserver_interface=127.21.66.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab
--principal=kudu/127.21.66.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39913,127.21.66.125:33051,127.21.66.124:36059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:06:06.178030 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } attempt: 1
I20260620 23:06:06.178681 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.178528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:43002)
0620 23:06:06.178600 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.178648 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:06:06.179098 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.179004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:43018)
0620 23:06:06.179055 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.179081 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 23:06:06.179222 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:33051: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
W20260620 23:06:06.224762 11960 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:06:06.224995 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:06.225021 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:06.225044 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:06.226272 11960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:06.226315 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:06.226329 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:06.226342 11960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:06.226397 11960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:06.227871 11960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39913,127.21.66.125:33051,127.21.66.124:36059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.125.keytab
--principal=kudu/127.21.66.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.125:33051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.66.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:06.228139 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:06.228376 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:06.233383 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:06.233451 11960 server_base.cc:1034] running on GCE node
W20260620 23:06:06.233395 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:06.233640 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:06.233886 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:06.234273 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:06.235424 11960 hybrid_clock.cc:648] HybridClock initialized: now 1781996766235418 us; error 40 us; skew 500 ppm
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:06.237965 11960 init.cc:377] Logged in from keytab as kudu/127.21.66.125@KRBTEST.COM (short username kudu)
I20260620 23:06:06.238504 11960 webserver.cc:460] Webserver started at http://127.21.66.125:35217/ using document root <none> and password file <none>
I20260620 23:06:06.238735 11960 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:06.238781 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:06.238929 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:06.240289 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data/instance:
uuid: "2069bbb602444a54815736188e201198"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "fba26b010be31c8be815a6cff3fdde0c"
server_key_iv: "2dc03d781a268f874fd1b80d23774034"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.240670 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/wal/instance:
uuid: "2069bbb602444a54815736188e201198"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "fba26b010be31c8be815a6cff3fdde0c"
server_key_iv: "2dc03d781a268f874fd1b80d23774034"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.242219 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 23:06:06.243021 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:06.243207 11960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:06:06.243317 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/wal
uuid: "2069bbb602444a54815736188e201198"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "fba26b010be31c8be815a6cff3fdde0c"
server_key_iv: "2dc03d781a268f874fd1b80d23774034"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.243392 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:06.247460 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } attempt: 2
I20260620 23:06:06.247903 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.247767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:43022)
0620 23:06:06.247832 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.247873 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 23:06:06.248273 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.248166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:43038)
0620 23:06:06.248215 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.248245 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:06:06.248365 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.125:33051: Network error: Client connection negotiation failed: client connection to 127.21.66.125:33051: connect: Connection refused (error 111)
I20260620 23:06:06.257846 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:06.260278 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:06.260447 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:06.264787 11960 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.125:33051
I20260620 23:06:06.264843 12027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.125:33051 every 8 connection(s)
I20260620 23:06:06.265232 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/data/info.pb
I20260620 23:06:06.266350 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:06.266832 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 11960
I20260620 23:06:06.266956 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-1/wal/instance
I20260620 23:06:06.267132 21769 external_mini_cluster.cc:1428] Setting key d188412b21c936a1c23f8ce5d9d7f426
I20260620 23:06:06.268163 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.66.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.66.124@KRBTEST.COM" created.
I20260620 23:06:06.278641 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.268986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.21.66.126:39913)
0620 23:06:06.269107 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.269109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.269304 (+ 195us) server_negotiation.cc:407] Connection header received
0620 23:06:06.270184 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.270185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.270203 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.270278 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.271205 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.271704 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.272789 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.273004 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.273104 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.275583 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.275585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.275585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.275593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.277123 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.277540 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.277541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.277542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.277617 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.277898 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.277898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.277899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.278022 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.278164 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.278446 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.278533 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 23:06:06.278874 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.268953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39913 (local address 127.0.0.1:33824)
0620 23:06:06.269173 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.269211 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.270018 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.270368 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.270373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.270791 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.271099 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.271106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.271808 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.271809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.272683 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.272686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.273125 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.273127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.273260 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.273857 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.273865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.275438 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.277198 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.277205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.277207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.277434 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.277685 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.277686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.277687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.277839 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.278153 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.278160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.278310 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.278424 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.278632 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260620 23:06:06.279697 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:06:06.287632 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.279981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:43048)
0620 23:06:06.280034 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.280113 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.280152 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.280435 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.280436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.280637 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.280702 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.280704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.281740 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.281741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.282317 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.282318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.282796 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.282797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.282860 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.283384 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.283389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.284662 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.286486 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.286487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.286488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.286659 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.286889 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.286889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.286890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.286944 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.287291 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.287292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.287329 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.287400 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.287502 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 23:06:06.287650 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.280118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.21.66.125:33051)
0620 23:06:06.280275 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.280281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.280305 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:06:06.280329 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.280332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.280350 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.280423 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.280784 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.281634 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.282394 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.282707 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.282799 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.284768 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.284784 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.284784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.284791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.286396 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.286754 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.286757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.286758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.286813 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.287009 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.287010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.287010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.287217 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.287314 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.287430 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.287555 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260620 23:06:06.288365 12028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:06.288786 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.288656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53246)
0620 23:06:06.288702 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.288745 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 23:06:06.288970 11976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
I20260620 23:06:06.289459 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.289370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53250)
0620 23:06:06.289418 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.289441 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 23:06:06.289589 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36059: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.66.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab.
Entry for principal kudu/127.21.66.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab.
I20260620 23:06:06.292881 21769 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxQfmfq/build/release/bin/kudu
/tmp/dist-test-taskxQfmfq/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:36059
--webserver_interface=127.21.66.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab
--principal=kudu/127.21.66.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.66.84:41705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.66.126:39913,127.21.66.125:33051,127.21.66.124:36059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:06:06.338800 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } attempt: 1
I20260620 23:06:06.339386 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.339199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53260)
0620 23:06:06.339296 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.339350 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 23:06:06.339519 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } attempt: 3
I20260620 23:06:06.339854 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.339734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53272)
0620 23:06:06.339787 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.339822 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:06:06.340025 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36059: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:06:06.347299 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.339846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.21.66.125:33051)
0620 23:06:06.339879 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.339881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.339891 (+ 10us) server_negotiation.cc:407] Connection header received
0620 23:06:06.340088 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.340089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.340105 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.340164 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.340517 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.341042 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.341771 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.342023 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.342141 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.344118 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.344120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.344120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.344129 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.345791 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.346353 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.346354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.346355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.346422 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.346672 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.346673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.346674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.346821 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.346887 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.347118 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.347220 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 23:06:06.347301 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.339781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:43056)
0620 23:06:06.339828 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.339839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.339911 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.340171 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.340173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.340338 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.340405 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.340406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.341138 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.341144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.341684 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.341685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.342135 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.342136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.342218 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.342730 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.342735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.343971 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.345896 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.345897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.345898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.346276 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.346531 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.346532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.346532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.346599 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.346918 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.346920 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.346974 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.347055 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.347195 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 23:06:06.347883 11953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:06:06.348268 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.348152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53286)
0620 23:06:06.348201 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.348239 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 23:06:06.348626 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.348525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53294)
0620 23:06:06.348579 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.348606 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 23:06:06.348748 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36059: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
W20260620 23:06:06.377185 12035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:06:06.377455 12035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:06:06.377486 12035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:06:06.377542 12035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:06:06.378726 12035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:06:06.378798 12035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:06:06.378821 12035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:06:06.378840 12035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:06:06.378872 12035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:06:06.379897 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } attempt: 1
I20260620 23:06:06.380403 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.380273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53298)
0620 23:06:06.380335 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.380376 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 23:06:06.380692 12035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.66.84:41705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.66.126:39913,127.21.66.125:33051,127.21.66.124:36059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/krb5kdc/kudu_127.21.66.124.keytab
--principal=kudu/127.21.66.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.66.124:36059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.66.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 22:02:53 UTC on ccycloud.quasar-fhklyl.root.comops.site
build id 4457793
I20260620 23:06:06.380740 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.380644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53300)
0620 23:06:06.380692 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.380721 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 23:06:06.380820 11953 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36059: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
I20260620 23:06:06.380987 12035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:06:06.381227 12035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:06:06.386246 12040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:06.386240 12043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:06:06.386250 12041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:06:06.386509 12035 server_base.cc:1034] running on GCE node
I20260620 23:06:06.386766 12035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:06:06.387022 12035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:06:06.388180 12035 hybrid_clock.cc:648] HybridClock initialized: now 1781996766388160 us; error 32 us; skew 500 ppm
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:06:06.390919 12035 init.cc:377] Logged in from keytab as kudu/127.21.66.124@KRBTEST.COM (short username kudu)
I20260620 23:06:06.391403 12035 webserver.cc:460] Webserver started at http://127.21.66.124:45379/ using document root <none> and password file <none>
I20260620 23:06:06.391649 12035 fs_manager.cc:359] Metadata directory not provided
I20260620 23:06:06.391695 12035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:06:06.391868 12035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:06:06.393006 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data/instance:
uuid: "681beb73c7894d8c994fc75c83b59b67"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "86f056d3f26ebfa94321544ce8096496"
server_key_iv: "52037a16d7e54b7128294a5ad35a979a"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.393385 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/wal/instance:
uuid: "681beb73c7894d8c994fc75c83b59b67"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "86f056d3f26ebfa94321544ce8096496"
server_key_iv: "52037a16d7e54b7128294a5ad35a979a"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.394867 12035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 23:06:06.395735 12050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:06:06.395939 12035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 23:06:06.396044 12035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data,/tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/wal
uuid: "681beb73c7894d8c994fc75c83b59b67"
format_stamp: "Formatted at 2026-06-20 23:06:06 on dist-test-slave-qgtk"
server_key: "86f056d3f26ebfa94321544ce8096496"
server_key_iv: "52037a16d7e54b7128294a5ad35a979a"
server_key_version: "encryptionkey@0"
I20260620 23:06:06.396130 12035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:06:06.408182 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } attempt: 2
I20260620 23:06:06.408618 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.408475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53316)
0620 23:06:06.408540 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.408579 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 23:06:06.409125 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.409017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53322)
0620 23:06:06.409072 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.409102 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 23:06:06.409283 12028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.66.124:36059: Network error: Client connection negotiation failed: client connection to 127.21.66.124:36059: connect: Connection refused (error 111)
I20260620 23:06:06.409962 12035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:06:06.412124 12035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:06:06.412238 12035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:06:06.416491 12035 rpc_server.cc:307] RPC server started. Bound to: 127.21.66.124:36059
I20260620 23:06:06.416524 12102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.66.124:36059 every 8 connection(s)
I20260620 23:06:06.416951 12035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/data/info.pb
I20260620 23:06:06.418044 12103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:06:06.418231 21769 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxQfmfq/build/release/bin/kudu as pid 12035
I20260620 23:06:06.418370 21769 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxQfmfq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781996487602640-21769-0/minicluster-data/master-2/wal/instance
I20260620 23:06:06.418561 21769 external_mini_cluster.cc:1428] Setting key acda7cf9d8449583690b7e66c2234ebc
I20260620 23:06:06.420871 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:06:06.430469 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.421620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.21.66.126:39913)
0620 23:06:06.421690 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.421692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.421858 (+ 166us) server_negotiation.cc:407] Connection header received
0620 23:06:06.421912 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.421913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.421923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.421989 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.422659 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.423200 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.424148 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.424429 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.424485 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.427511 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.427512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.427512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.427519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.428952 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.429351 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.429361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.429362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.429428 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.429660 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.429661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.429661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.429813 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.429900 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.430275 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.430373 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 23:06:06.430719 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.421529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.126:39913 (local address 127.0.0.1:33828)
0620 23:06:06.421722 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.421776 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.421866 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.422033 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.422036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.422481 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.422571 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.422574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.423291 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.423298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.424063 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.424072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.424548 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.424550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.424636 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.425076 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.425086 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.427417 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.429039 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.429045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.429047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.429268 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.429493 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.429495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.429495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.429594 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.429891 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.429897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.430067 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.430245 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.430456 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":118,"threads_started":1}
I20260620 23:06:06.430721 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.419115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.21.66.124:36059)
0620 23:06:06.419298 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.419306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.419332 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:06:06.419802 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.419809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.419861 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.419980 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.420880 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.422022 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.423097 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.423600 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.423684 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.427177 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.427187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.427188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.427195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.428979 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.429348 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.429351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.429351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.429440 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.429883 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.429884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.429884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.430070 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.430273 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.430398 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.430515 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260620 23:06:06.431423 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.126@KRBTEST.COM
I20260620 23:06:06.439275 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.431854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.21.66.125:33051)
0620 23:06:06.431902 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.431904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.431911 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:06:06.431931 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.431931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.431939 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.431978 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.432315 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.432955 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.433811 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.434107 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.434210 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.436287 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.436289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.436290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.436298 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.437932 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.438357 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.438358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.438358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.438414 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.438696 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.438697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.438697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.438838 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.438924 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.439091 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.439174 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 23:06:06.439275 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.431687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.125:33051 (local address 127.0.0.1:43062)
0620 23:06:06.431730 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.431739 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.431771 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.432013 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.432014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.432188 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.432235 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.432237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.433039 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.433040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.433733 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.433735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.434229 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.434230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.434296 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.434879 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.434885 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.436195 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.438031 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.438032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.438032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.438287 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.438506 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.438507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.438507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.438628 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.438917 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.438918 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.438975 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.439065 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.439173 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 23:06:06.440029 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.431803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.21.66.126:39913)
0620 23:06:06.431980 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.431982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.431990 (+ 8us) server_negotiation.cc:407] Connection header received
0620 23:06:06.432012 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.432012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.432021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.432080 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.432665 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.433365 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.434223 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.434506 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.434576 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.437010 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.437011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.437011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.437019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.438821 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.439179 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.439180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.439181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.439237 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.439449 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.439450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.439454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.439576 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.439723 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.439835 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.439922 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260620 23:06:06.440125 12103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.124@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.66.125@KRBTEST.COM
I20260620 23:06:06.447881 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.440390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53342)
0620 23:06:06.440426 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.440439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.440481 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.440681 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.440682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.440861 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.440906 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.440908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.441677 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.441678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.442361 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.442363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.442826 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.442827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.442887 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.443394 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.443398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.444824 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.446656 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.446657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.446658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.446848 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.447215 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.447216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.447216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.447280 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.447537 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.447538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.447576 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.447644 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.447755 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":6}
I20260620 23:06:06.447887 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.440526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.21.66.124:36059)
0620 23:06:06.440559 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.440561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.440566 (+ 5us) server_negotiation.cc:407] Connection header received
0620 23:06:06.440581 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.440582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.440590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.440636 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.440988 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.441586 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.442532 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.442746 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.442831 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.444912 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.444914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.444914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.444923 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.446576 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.447022 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.447023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.447023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.447092 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.447345 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.447347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.447347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.447468 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.447545 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.447668 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.447803 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":19}
I20260620 23:06:06.449221 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.440933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.21.66.125:33051)
0620 23:06:06.440990 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.440992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.441096 (+ 104us) server_negotiation.cc:407] Connection header received
0620 23:06:06.441137 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.441138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.441147 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.441213 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.441709 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.442232 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.443145 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.443459 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.443514 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.446498 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.446500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.446500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.446508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.448122 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.448519 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.448519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.448520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.448588 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.448797 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.448798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.448798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.448894 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.449003 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.449095 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.449157 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 23:06:06.449672 12103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: Bootstrap starting.
I20260620 23:06:06.450659 12103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:06.450994 12103 log.cc:824] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:06.451853 21769 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:06:06.451926 12103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: No bootstrap required, opened a new log
I20260620 23:06:06.452886 11927 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "04f339ea0f0047ed90d4a8c013104b38"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33838
I20260620 23:06:06.453866 12103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.454023 12103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681beb73c7894d8c994fc75c83b59b67, State: Initialized, Role: FOLLOWER
I20260620 23:06:06.454152 12103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.454432 12109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.454500 12103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:06.454551 12109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:06.455933 11953 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } attempt: 2
W20260620 23:06:06.456362 12120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:06:06.456426 12120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:06:06.456700 12035 master_runner.cc:186] Error getting master registration for 127.21.66.126:39913: OK, instance_id { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" instance_seqno: 1781996766111049 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:06:06.459847 12035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.126@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
I20260620 23:06:06.464222 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.456411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.21.66.124:36059)
0620 23:06:06.456447 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.456448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.456455 (+ 7us) server_negotiation.cc:407] Connection header received
0620 23:06:06.456474 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.456474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.456483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.456528 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.456884 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.457586 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.458647 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.458921 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.458984 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.461300 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.461301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.461301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.461308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.462812 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.463263 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.463264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.463264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.463320 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.463560 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.463561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.463561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.463740 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.463816 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.464037 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.464124 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:06:06.464232 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.456201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53354)
0620 23:06:06.456283 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.456294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.456333 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.456572 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.456573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.456752 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.456799 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.456801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.457685 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.457687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.458553 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.458556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.459038 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.459039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.459094 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.459602 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.459617 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.461211 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.462880 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.462881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.462881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.463118 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.463395 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.463396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.463396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.463482 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.463854 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.463855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.463911 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.463994 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.464124 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 23:06:06.466223 11953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38: Bootstrap starting.
I20260620 23:06:06.466984 11953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:06.467308 11953 log.cc:824] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:06.468145 11953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38: No bootstrap required, opened a new log
I20260620 23:06:06.469525 11953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.469673 11953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f339ea0f0047ed90d4a8c013104b38, State: Initialized, Role: FOLLOWER
I20260620 23:06:06.469774 11953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.470099 11953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:06:06.470156 11927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:06:06.470199 11927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:06.471087 11927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.471388 11927 leader_election.cc:288] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [CANDIDATE]: Term 1 election: Requested vote from peers 2069bbb602444a54815736188e201198 (127.21.66.125:33051), 681beb73c7894d8c994fc75c83b59b67 (127.21.66.124:36059)
I20260620 23:06:06.471781 12077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04f339ea0f0047ed90d4a8c013104b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "681beb73c7894d8c994fc75c83b59b67"
I20260620 23:06:06.471927 12077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:06.470069 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.472519 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:06.472848 12077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04f339ea0f0047ed90d4a8c013104b38 in term 1.
I20260620 23:06:06.473071 11901 leader_election.cc:302] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04f339ea0f0047ed90d4a8c013104b38, 681beb73c7894d8c994fc75c83b59b67; no voters:
I20260620 23:06:06.473220 12123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:06:06.473358 12123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [term 1 LEADER]: Becoming Leader. State: Replica: 04f339ea0f0047ed90d4a8c013104b38, State: Running, Role: LEADER
I20260620 23:06:06.473452 12123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.473894 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f339ea0f0047ed90d4a8c013104b38. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.473958 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:06.474144 12143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:06:06.474610 12143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:06:06.473013 11885 master_runner.cc:186] Error getting master registration for 127.21.66.125:33051: OK, instance_id { permanent_uuid: "2069bbb602444a54815736188e201198" instance_seqno: 1781996766263501 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:06:06.476617 11885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:06:06.485288 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.473119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.21.66.124:36059)
0620 23:06:06.473197 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.473199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.474484 (+ 1285us) server_negotiation.cc:407] Connection header received
0620 23:06:06.474573 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.474574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.474585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.474613 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.477107 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.477825 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.478846 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.479155 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.479215 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.481892 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.481893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.481893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.481901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.483663 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.484096 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.484097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.484097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.484154 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.484651 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.484652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.484653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.484800 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.484871 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.485080 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.485165 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 23:06:06.485526 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.472790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.21.66.125:33051)
0620 23:06:06.472834 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.472836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.473357 (+ 521us) server_negotiation.cc:407] Connection header received
0620 23:06:06.473475 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.473476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.473487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.473518 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.475317 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.476026 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.478867 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.479158 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.479201 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.480920 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.480922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.480922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.480931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.484097 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.484506 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.484507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.484508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.484566 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.484836 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.484837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.484838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.484979 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.485050 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.485349 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.485434 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 23:06:06.487303 12077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [term 1 FOLLOWER]: Refusing update from remote peer 04f339ea0f0047ed90d4a8c013104b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:06:06.487691 12123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:06:06.489562 12109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f339ea0f0047ed90d4a8c013104b38. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.489656 12109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:06.492374 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.492480 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:06.492987 12109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.493073 12109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:06.493278 12147 mvcc.cc:205] Tried to move back new op lower bound from 7299058755528974336 to 7299058755476115456. Current Snapshot: MvccSnapshot[applied={T|T < 7299058755528974336}]
I20260620 23:06:06.493451 12145 mvcc.cc:205] Tried to move back new op lower bound from 7299058755528974336 to 7299058755476115456. Current Snapshot: MvccSnapshot[applied={T|T < 7299058755528974336}]
I20260620 23:06:06.494104 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.476910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.21.66.126:39913)
0620 23:06:06.476948 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.476949 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.485852 (+ 8903us) server_negotiation.cc:407] Connection header received
0620 23:06:06.485944 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.485945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.485956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.485985 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.486647 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.487386 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.488317 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.488580 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.488626 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.490175 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.490176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.490176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.490184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.491850 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.492602 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.492603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.492603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.492678 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.493237 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.493238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.493239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.493384 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.493467 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.493917 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.494010 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:06:06.494864 12143 catalog_manager.cc:1347] Generated new cluster ID: 315ed7ce8c074a1bb374126bf57f699f
I20260620 23:06:06.494921 12143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:06:06.500406 12028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } attempt: 3
Jun 20 23:06:06 dist-test-slave-qgtk krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781996766, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.66.125@KRBTEST.COM for kudu/127.21.66.124@KRBTEST.COM
I20260620 23:06:06.508853 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.500733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.66.124:36059 (local address 127.0.0.1:53372)
0620 23:06:06.500788 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 23:06:06.500802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 23:06:06.500851 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:06:06.500992 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:06:06.500993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:06:06.501185 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:06:06.501245 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.501247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.502186 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.502188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.503079 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:06:06.503083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.503615 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.503616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:06:06.503672 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.504155 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:06:06.504160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:06:06.505746 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:06:06.507773 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.507774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.507774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.507989 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.508185 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:06:06.508186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:06:06.508186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:06:06.508239 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:06:06.508528 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:06:06.508529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:06:06.508577 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:06:06.508639 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 23:06:06.508743 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 23:06:06.509199 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:06.500831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.21.66.124:36059)
0620 23:06:06.500870 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.500872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.500881 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:06:06.500908 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.500909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.500919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.500977 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.501317 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.502062 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.503184 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:06.503499 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:06.503556 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:06.505853 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:06:06.505855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:06:06.505855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:06:06.505863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:06:06.507673 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.508067 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.508068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.508068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.508121 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:06:06.508304 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:06:06.508304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:06:06.508305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:06:06.508444 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:06:06.508503 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:06.509036 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 23:06:06.509115 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:06:06.510890 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: Bootstrap starting.
I20260620 23:06:06.511632 12028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: Neither blocks nor log segments found. Creating new log.
I20260620 23:06:06.511961 12028 log.cc:824] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: Log is configured to *not* fsync() on all Append() calls
I20260620 23:06:06.523078 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: No bootstrap required, opened a new log
I20260620 23:06:06.525631 12028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.525945 12028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2069bbb602444a54815736188e201198, State: Initialized, Role: FOLLOWER
I20260620 23:06:06.526183 12028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } }
I20260620 23:06:06.527067 12001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:06:06.528067 12001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [term 1 FOLLOWER]: Refusing update from remote peer 04f339ea0f0047ed90d4a8c013104b38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:06:06.528175 12143 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:06:06.528370 12123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:06:06.528918 12143 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:06:06.527094 12002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04f339ea0f0047ed90d4a8c013104b38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2069bbb602444a54815736188e201198"
I20260620 23:06:06.526954 12028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:06:06.533521 12167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:06:06.533638 12167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:06:06.533825 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.533987 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:06.534073 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f339ea0f0047ed90d4a8c013104b38. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.534190 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:06.534257 12153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.534422 12153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:06:06.534966 12154 mvcc.cc:205] Tried to move back new op lower bound from 7299058755693383680 to 7299058755476115456. Current Snapshot: MvccSnapshot[applied={T|T < 7299058755528974336}]
I20260620 23:06:06.535812 12002 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 04f339ea0f0047ed90d4a8c013104b38 for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 23:06:06.536515 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04f339ea0f0047ed90d4a8c013104b38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04f339ea0f0047ed90d4a8c013104b38" member_type: VOTER last_known_addr { host: "127.21.66.126" port: 39913 } } peers { permanent_uuid: "2069bbb602444a54815736188e201198" member_type: VOTER last_known_addr { host: "127.21.66.125" port: 33051 } } peers { permanent_uuid: "681beb73c7894d8c994fc75c83b59b67" member_type: VOTER last_known_addr { host: "127.21.66.124" port: 36059 } } }
I20260620 23:06:06.536615 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38 [sys.catalog]: This master's current role is: LEADER
I20260620 23:06:06.537564 12143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04f339ea0f0047ed90d4a8c013104b38: Generated new TSK 0
I20260620 23:06:06.537784 12143 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:06:06.568686 11957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.567564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.21.66.126:39913)
0620 23:06:06.567613 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.567616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.568000 (+ 384us) server_negotiation.cc:407] Connection header received
0620 23:06:06.568110 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.568111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.568124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.568155 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.568613 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33864: BlockingRecv error: recv got EOF from 127.0.0.1:33864 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 23:06:06.570025 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.569059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.21.66.125:33051)
0620 23:06:06.569100 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.569102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.569433 (+ 331us) server_negotiation.cc:407] Connection header received
0620 23:06:06.569555 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.569556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.569567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.569596 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.569945 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43100: BlockingRecv error: recv got EOF from 127.0.0.1:43100 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:06:06.570904 12106 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.569293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.21.66.124:36059)
0620 23:06:06.569333 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.569334 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.570330 (+ 996us) server_negotiation.cc:407] Connection header received
0620 23:06:06.570427 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.570428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.570439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.570468 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.570827 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53388: BlockingRecv error: recv got EOF from 127.0.0.1:53388 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:06:06.573231 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.572459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.21.66.125:33051)
0620 23:06:06.572495 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.572497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.572680 (+ 183us) server_negotiation.cc:407] Connection header received
0620 23:06:06.572790 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.572791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.572810 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.572840 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.573183 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43108: BlockingRecv error: recv got EOF from 127.0.0.1:43108 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 23:06:06.576273 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.575520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.21.66.125:33051)
0620 23:06:06.575554 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.575556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.575765 (+ 209us) server_negotiation.cc:407] Connection header received
0620 23:06:06.575862 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.575863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.575888 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.575918 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.576229 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43122: BlockingRecv error: recv got EOF from 127.0.0.1:43122 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 23:06:06.580219 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.579431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.21.66.125:33051)
0620 23:06:06.579466 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.579468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.579697 (+ 229us) server_negotiation.cc:407] Connection header received
0620 23:06:06.579794 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.579795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.579806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.579836 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.580175 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43136: BlockingRecv error: recv got EOF from 127.0.0.1:43136 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 23:06:06.586347 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.585410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.21.66.125:33051)
0620 23:06:06.585447 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.585449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.585667 (+ 218us) server_negotiation.cc:407] Connection header received
0620 23:06:06.585768 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.585769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.585781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.585812 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.586300 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43148: BlockingRecv error: recv got EOF from 127.0.0.1:43148 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:06:06.596529 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.595691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.21.66.125:33051)
0620 23:06:06.595730 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.595732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.595970 (+ 238us) server_negotiation.cc:407] Connection header received
0620 23:06:06.596074 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.596075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.596088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.596118 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.596478 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43164: BlockingRecv error: recv got EOF from 127.0.0.1:43164 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 23:06:06.614866 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.613941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.21.66.125:33051)
0620 23:06:06.613989 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.613991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.614285 (+ 294us) server_negotiation.cc:407] Connection header received
0620 23:06:06.614384 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.614385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.614396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.614424 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.614806 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43176: BlockingRecv error: recv got EOF from 127.0.0.1:43176 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 23:06:06.648983 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.648205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.21.66.125:33051)
0620 23:06:06.648253 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.648255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.648416 (+ 161us) server_negotiation.cc:407] Connection header received
0620 23:06:06.648490 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.648491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.648504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.648556 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.648909 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43192: BlockingRecv error: recv got EOF from 127.0.0.1:43192 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 23:06:06.715121 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.714228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.21.66.125:33051)
0620 23:06:06.714333 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.714336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.714476 (+ 140us) server_negotiation.cc:407] Connection header received
0620 23:06:06.714604 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.714605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.714615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.714687 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.715056 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43194: BlockingRecv error: recv got EOF from 127.0.0.1:43194 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 23:06:06.845384 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:06.844489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.21.66.125:33051)
0620 23:06:06.844565 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:06.844568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:06.844773 (+ 205us) server_negotiation.cc:407] Connection header received
0620 23:06:06.844808 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:06.844809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:06.844821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:06.844889 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:06.845313 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43208: BlockingRecv error: recv got EOF from 127.0.0.1:43208 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 23:06:07.103511 12033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:06:07.102804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.21.66.125:33051)
0620 23:06:07.102889 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.102891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.102900 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:06:07.102967 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.102968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.102978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.103036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:06:07.103421 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43224: BlockingRecv error: recv got EOF from 127.0.0.1:43224 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 23:06:07.456883 12120 catalog_manager.cc:1259] Loaded cluster ID: 315ed7ce8c074a1bb374126bf57f699f
I20260620 23:06:07.457087 12120 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: loading cluster ID for follower catalog manager: success
I20260620 23:06:07.457942 12120 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: acquiring CA information for follower catalog manager: success
I20260620 23:06:07.458294 12120 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 681beb73c7894d8c994fc75c83b59b67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:06:07.534219 12167 catalog_manager.cc:1259] Loaded cluster ID: 315ed7ce8c074a1bb374126bf57f699f
I20260620 23:06:07.534296 12167 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: loading cluster ID for follower catalog manager: success
I20260620 23:06:07.535184 12167 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: acquiring CA information for follower catalog manager: success
I20260620 23:06:07.535518 12167 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2069bbb602444a54815736188e201198: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:06:07.619835 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.616811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.21.66.125:33051)
0620 23:06:07.616974 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.616976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.617004 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:06:07.617081 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.617082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.617132 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.617211 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:07.617348 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.617924 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.618930 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.619133 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.619197 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.619370 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:07.619482 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:07.619544 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:07.619657 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 23:06:07.619677 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260620 23:06:07.624338 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.621789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.21.66.124:36059)
0620 23:06:07.621907 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.621910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.621919 (+ 9us) server_negotiation.cc:407] Connection header received
0620 23:06:07.622013 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.622013 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.622049 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.622174 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:07.622293 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.622798 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.623541 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.623743 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.623818 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.623990 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:07.624093 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:07.624182 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:07.624221 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 23:06:07.624242 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260620 23:06:07.629108 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.626028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.21.66.125:33051)
0620 23:06:07.626082 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.626084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.626348 (+ 264us) server_negotiation.cc:407] Connection header received
0620 23:06:07.626446 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.626448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.626460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.626517 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:07.626855 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.627601 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.628337 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.628568 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.628628 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.628795 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:07.628830 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:07.628884 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:07.629011 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 23:06:07.629035 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 23:06:07.629321 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.626320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.21.66.124:36059)
0620 23:06:07.626357 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.626359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.626437 (+ 78us) server_negotiation.cc:407] Connection header received
0620 23:06:07.626577 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.626578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.626604 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.626665 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:07.626815 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.627543 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.628543 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.628820 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.628887 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.629025 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:07.629053 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:07.629139 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:07.629241 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 23:06:07.629263 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 23:06:07.629554 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:06:07.626048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.21.66.126:39913)
0620 23:06:07.626189 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 23:06:07.626191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 23:06:07.626408 (+ 217us) server_negotiation.cc:407] Connection header received
0620 23:06:07.626457 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:06:07.626457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:06:07.626543 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:06:07.626641 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:06:07.626793 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.627309 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.628365 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:06:07.628684 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:06:07.628752 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:06:07.629082 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:06:07.629195 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:06:07.629257 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 23:06:07.629448 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 23:06:07.629470 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260620 23:06:07.635138 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11885
I20260620 23:06:07.640924 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 11960
I20260620 23:06:07.646020 21769 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxQfmfq/build/release/bin/kudu with pid 12035
2026-06-20T23:06:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2825 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5657 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280050 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST